2024-03-29T11:37:15.961 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T11:37:15.962 DEBUG:teuthology.run:Teuthology command: teuthology --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} --owner scheduled_amathuri@teuthology --verbose --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622735 -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622735/orig.config.yaml 2024-03-29T11:37:15.997 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:37:16.081 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622735 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622735' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi098.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFYgWZ1LbgwbHhpk7VTsXP9V/QXU8ImHq/fUj86pKZUcizIBTgap0pTq4nwuQdS5UBBTwAE0Tft+80B3oG7NyjA= smithi121.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFEeDHpd8zmJPFCJ6dqE7CuXyzXFk53/XMEvbQKq08tzZzHUQC4pkdpPPyFRqyFEUI2I65wyAYFYYTc38KvJnzE= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T11:37:16.081 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T11:37:16.082 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T11:37:16.082 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T11:37:16.084 INFO:teuthology.task.internal:Checking packages... 2024-03-29T11:37:16.109 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T11:37:16.109 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T11:37:16.109 INFO:teuthology.packaging:ref: None 2024-03-29T11:37:16.109 INFO:teuthology.packaging:tag: None 2024-03-29T11:37:16.110 INFO:teuthology.packaging:branch: main 2024-03-29T11:37:16.110 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:37:16.110 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T11:37:16.381 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T11:37:16.382 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T11:37:16.393 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T11:37:16.393 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T11:37:16.406 INFO:teuthology.task.internal:Saving configuration 2024-03-29T11:37:16.420 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T11:37:16.430 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:37:16.457 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622735', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:32:11.731821', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFYgWZ1LbgwbHhpk7VTsXP9V/QXU8ImHq/fUj86pKZUcizIBTgap0pTq4nwuQdS5UBBTwAE0Tft+80B3oG7NyjA='} 2024-03-29T11:37:16.482 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622735', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:32:11.733908', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFEeDHpd8zmJPFCJ6dqE7CuXyzXFk53/XMEvbQKq08tzZzHUQC4pkdpPPyFRqyFEUI2I65wyAYFYYTc38KvJnzE='} 2024-03-29T11:37:16.482 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T11:37:16.493 INFO:teuthology.task.internal:roles: ubuntu@smithi098.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T11:37:16.494 INFO:teuthology.task.internal:roles: ubuntu@smithi121.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T11:37:16.494 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T11:37:16.572 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1bcc0750d0>, signals=[15]) 2024-03-29T11:37:16.573 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T11:37:16.591 INFO:teuthology.task.internal:Opening connections... 2024-03-29T11:37:16.591 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi098.front.sepia.ceph.com 2024-03-29T11:37:16.593 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:37:16.685 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi121.front.sepia.ceph.com 2024-03-29T11:37:16.685 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:37:16.773 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T11:37:16.781 DEBUG:teuthology.orchestra.run.smithi098:> uname -m 2024-03-29T11:37:16.798 INFO:teuthology.orchestra.run.smithi098.stdout:x86_64 2024-03-29T11:37:16.798 DEBUG:teuthology.orchestra.run.smithi098:> cat /etc/os-release 2024-03-29T11:37:16.853 INFO:teuthology.orchestra.run.smithi098.stdout:NAME="CentOS Stream" 2024-03-29T11:37:16.854 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION="9" 2024-03-29T11:37:16.854 INFO:teuthology.orchestra.run.smithi098.stdout:ID="centos" 2024-03-29T11:37:16.854 INFO:teuthology.orchestra.run.smithi098.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:37:16.854 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_ID="9" 2024-03-29T11:37:16.854 INFO:teuthology.orchestra.run.smithi098.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:37:16.854 INFO:teuthology.orchestra.run.smithi098.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:37:16.854 INFO:teuthology.orchestra.run.smithi098.stdout:ANSI_COLOR="0;31" 2024-03-29T11:37:16.854 INFO:teuthology.orchestra.run.smithi098.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:37:16.854 INFO:teuthology.orchestra.run.smithi098.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:37:16.854 INFO:teuthology.orchestra.run.smithi098.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:37:16.854 INFO:teuthology.orchestra.run.smithi098.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:37:16.854 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:37:16.854 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:37:16.855 INFO:teuthology.lock.ops:Updating smithi098.front.sepia.ceph.com on lock server 2024-03-29T11:37:16.872 DEBUG:teuthology.orchestra.run.smithi121:> uname -m 2024-03-29T11:37:16.888 INFO:teuthology.orchestra.run.smithi121.stdout:x86_64 2024-03-29T11:37:16.889 DEBUG:teuthology.orchestra.run.smithi121:> cat /etc/os-release 2024-03-29T11:37:16.944 INFO:teuthology.orchestra.run.smithi121.stdout:NAME="CentOS Stream" 2024-03-29T11:37:16.944 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION="9" 2024-03-29T11:37:16.944 INFO:teuthology.orchestra.run.smithi121.stdout:ID="centos" 2024-03-29T11:37:16.944 INFO:teuthology.orchestra.run.smithi121.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:37:16.945 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_ID="9" 2024-03-29T11:37:16.945 INFO:teuthology.orchestra.run.smithi121.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:37:16.945 INFO:teuthology.orchestra.run.smithi121.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:37:16.945 INFO:teuthology.orchestra.run.smithi121.stdout:ANSI_COLOR="0;31" 2024-03-29T11:37:16.945 INFO:teuthology.orchestra.run.smithi121.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:37:16.945 INFO:teuthology.orchestra.run.smithi121.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:37:16.945 INFO:teuthology.orchestra.run.smithi121.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:37:16.945 INFO:teuthology.orchestra.run.smithi121.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:37:16.945 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:37:16.945 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:37:16.945 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server 2024-03-29T11:37:16.963 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T11:37:16.974 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T11:37:16.985 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T11:37:16.985 DEBUG:teuthology.orchestra.run.smithi098:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:37:16.988 DEBUG:teuthology.orchestra.run.smithi121:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:37:17.003 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T11:37:17.014 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T11:37:17.014 DEBUG:teuthology.orchestra.run.smithi098:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:37:17.044 DEBUG:teuthology.orchestra.run.smithi121:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:37:17.074 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T11:37:17.133 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T11:37:17.153 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T11:37:17.153 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T11:37:17.153 DEBUG:teuthology.orchestra.run.smithi098:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:37:17.154 DEBUG:teuthology.orchestra.run.smithi121:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:37:17.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:37:17.169 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2024-03-29T11:37:17.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:37:17.170 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-03-29T11:37:17.225 INFO:teuthology.orchestra.run.smithi121.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:37:17.226 INFO:teuthology.task.kernel:Running kernel on smithi121: 5.14.0-361.el9.x86_64 2024-03-29T11:37:17.226 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum install -y kernel 2024-03-29T11:37:17.227 INFO:teuthology.orchestra.run.smithi098.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:37:17.227 INFO:teuthology.task.kernel:Running kernel on smithi098: 5.14.0-361.el9.x86_64 2024-03-29T11:37:17.227 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum install -y kernel 2024-03-29T11:37:19.345 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-29T11:37:20.378 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-9 - AppStream 4.4 MB/s | 6.7 MB 00:01 2024-03-29T11:37:20.987 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-29T11:37:21.987 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-03-29T11:37:21.996 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-29T11:37:22.894 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-03-29T11:37:23.870 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-03-29T11:37:25.987 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 9.3 MB/s | 19 MB 00:02 2024-03-29T11:37:28.879 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 6.1 MB 00:01 2024-03-29T11:37:30.173 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 8.5 MB/s | 6.1 MB 00:00 2024-03-29T11:37:30.447 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-03-29T11:37:31.956 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 24 kB/s | 16 kB 00:00 2024-03-29T11:37:32.474 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-29T11:37:33.951 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-29T11:37:37.497 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-03-29T11:37:39.141 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 9.9 kB/s | 1.7 kB 00:00 2024-03-29T11:37:39.737 INFO:teuthology.orchestra.run.smithi121.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:37:39.737 INFO:teuthology.orchestra.run.smithi121.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:37:39.825 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:37:39.830 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:37:39.830 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-03-29T11:37:39.830 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:37:39.830 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-29T11:37:39.831 INFO:teuthology.orchestra.run.smithi121.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:37:39.831 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-03-29T11:37:39.831 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:37:39.831 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:37:39.831 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:37:39.831 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:37:39.831 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-29T11:37:39.831 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:37:39.831 INFO:teuthology.orchestra.run.smithi121.stdout:Install 4 Packages 2024-03-29T11:37:39.831 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:37:39.832 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 76 M 2024-03-29T11:37:39.832 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 123 M 2024-03-29T11:37:39.832 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-29T11:37:39.998 INFO:teuthology.orchestra.run.smithi121.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-03-29T11:37:40.690 INFO:teuthology.orchestra.run.smithi121.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 18 MB/s | 15 MB 00:00 2024-03-29T11:37:41.198 INFO:teuthology.orchestra.run.smithi121.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 25 MB/s | 34 MB 00:01 2024-03-29T11:37:41.422 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:37:41.423 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:37:41.509 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T11:37:41.515 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T11:37:41.515 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-29T11:37:41.515 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T11:37:41.516 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-29T11:37:41.516 INFO:teuthology.orchestra.run.smithi098.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:37:41.516 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-29T11:37:41.516 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:37:41.516 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:37:41.516 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:37:41.516 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:37:41.516 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-29T11:37:41.516 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T11:37:41.516 INFO:teuthology.orchestra.run.smithi098.stdout:Install 4 Packages 2024-03-29T11:37:41.516 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:37:41.517 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 76 M 2024-03-29T11:37:41.517 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 123 M 2024-03-29T11:37:41.517 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-29T11:37:41.582 INFO:teuthology.orchestra.run.smithi121.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 17 MB/s | 28 MB 00:01 2024-03-29T11:37:41.585 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:37:41.585 INFO:teuthology.orchestra.run.smithi121.stdout:Total 44 MB/s | 76 MB 00:01 2024-03-29T11:37:41.585 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-29T11:37:41.675 INFO:teuthology.orchestra.run.smithi098.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-29T11:37:41.945 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-29T11:37:41.945 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-29T11:37:42.434 INFO:teuthology.orchestra.run.smithi098.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 16 MB/s | 15 MB 00:00 2024-03-29T11:37:42.823 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-29T11:37:42.823 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-29T11:37:42.942 INFO:teuthology.orchestra.run.smithi098.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 24 MB/s | 34 MB 00:01 2024-03-29T11:37:43.209 INFO:teuthology.orchestra.run.smithi098.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 18 MB/s | 28 MB 00:01 2024-03-29T11:37:43.212 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:37:43.213 INFO:teuthology.orchestra.run.smithi098.stdout:Total 45 MB/s | 76 MB 00:01 2024-03-29T11:37:43.213 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-29T11:37:43.574 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-29T11:37:43.574 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-29T11:37:44.430 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-29T11:37:44.430 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-29T11:37:45.115 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-29T11:37:45.714 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:37:45.799 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:37:46.732 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-29T11:37:47.393 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:37:47.479 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:37:48.551 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:37:48.726 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:37:50.239 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:37:50.374 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:37:53.445 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:37:53.491 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:37:55.102 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:37:55.206 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:37:58.335 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:37:59.963 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:38:27.708 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:38:27.709 INFO:teuthology.orchestra.run.smithi121.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:38:27.709 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:38:27.751 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:38:28.082 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:38:28.082 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:38:28.082 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:38:28.082 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:38:28.536 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:38:28.536 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:38:28.536 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-29T11:38:28.536 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:38:28.536 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:38:28.536 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:38:28.536 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:38:28.536 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:38:28.537 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:38:28.837 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:38:28.944 INFO:teuthology.orchestra.run.smithi121.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:38:28.944 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:38:28.944 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:38:28.944 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:38:28.944 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:38:28.944 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T11:38:28.945 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi121.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:38:28.945 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:38:29.024 INFO:teuthology.orchestra.run.smithi121.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:38:29.025 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:38:29.025 DEBUG:teuthology.orchestra.run.smithi121:> sudo rpm -qi grub2-tools 2024-03-29T11:38:29.063 INFO:teuthology.orchestra.run.smithi121.stdout:Name : grub2-tools 2024-03-29T11:38:29.063 INFO:teuthology.orchestra.run.smithi121.stdout:Epoch : 1 2024-03-29T11:38:29.063 INFO:teuthology.orchestra.run.smithi121.stdout:Version : 2.06 2024-03-29T11:38:29.064 INFO:teuthology.orchestra.run.smithi121.stdout:Release : 68.el9 2024-03-29T11:38:29.064 INFO:teuthology.orchestra.run.smithi121.stdout:Architecture: x86_64 2024-03-29T11:38:29.064 INFO:teuthology.orchestra.run.smithi121.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:38:29.064 INFO:teuthology.orchestra.run.smithi121.stdout:Group : Unspecified 2024-03-29T11:38:29.064 INFO:teuthology.orchestra.run.smithi121.stdout:Size : 8264254 2024-03-29T11:38:29.064 INFO:teuthology.orchestra.run.smithi121.stdout:License : GPLv3+ 2024-03-29T11:38:29.064 INFO:teuthology.orchestra.run.smithi121.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:38:29.064 INFO:teuthology.orchestra.run.smithi121.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:38:29.064 INFO:teuthology.orchestra.run.smithi121.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:38:29.064 INFO:teuthology.orchestra.run.smithi121.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:38:29.064 INFO:teuthology.orchestra.run.smithi121.stdout:Packager : builder@centos.org 2024-03-29T11:38:29.064 INFO:teuthology.orchestra.run.smithi121.stdout:Vendor : CentOS 2024-03-29T11:38:29.064 INFO:teuthology.orchestra.run.smithi121.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:38:29.065 INFO:teuthology.orchestra.run.smithi121.stdout:Summary : Support tools for GRUB. 2024-03-29T11:38:29.065 INFO:teuthology.orchestra.run.smithi121.stdout:Description : 2024-03-29T11:38:29.065 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:38:29.065 INFO:teuthology.orchestra.run.smithi121.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:38:29.065 INFO:teuthology.orchestra.run.smithi121.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:38:29.065 INFO:teuthology.orchestra.run.smithi121.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:38:29.065 INFO:teuthology.orchestra.run.smithi121.stdout:hardware devices. 2024-03-29T11:38:29.065 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:38:29.065 INFO:teuthology.orchestra.run.smithi121.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:38:29.066 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:38:29.066 INFO:teuthology.task.kernel:Updating grub on smithi121 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:38:29.066 DEBUG:teuthology.orchestra.run.smithi121:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:38:29.389 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:38:29.389 INFO:teuthology.orchestra.run.smithi098.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:38:29.389 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:38:29.432 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:38:29.632 INFO:teuthology.orchestra.run.smithi121.stderr:Generating grub configuration file ... 2024-03-29T11:38:29.841 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:38:29.841 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:38:29.842 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:38:29.842 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:38:30.269 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:38:30.270 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:38:30.270 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-29T11:38:30.270 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:38:30.270 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:38:30.270 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:38:30.270 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:38:30.270 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:38:30.270 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T11:38:30.649 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:38:30.753 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:38:30.753 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:38:30.753 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:38:30.753 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:38:30.753 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:38:30.753 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T11:38:30.753 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi098.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:38:30.753 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:38:30.833 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:38:30.833 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:38:30.833 DEBUG:teuthology.orchestra.run.smithi098:> sudo rpm -qi grub2-tools 2024-03-29T11:38:30.867 INFO:teuthology.orchestra.run.smithi098.stdout:Name : grub2-tools 2024-03-29T11:38:30.867 INFO:teuthology.orchestra.run.smithi098.stdout:Epoch : 1 2024-03-29T11:38:30.867 INFO:teuthology.orchestra.run.smithi098.stdout:Version : 2.06 2024-03-29T11:38:30.867 INFO:teuthology.orchestra.run.smithi098.stdout:Release : 68.el9 2024-03-29T11:38:30.868 INFO:teuthology.orchestra.run.smithi098.stdout:Architecture: x86_64 2024-03-29T11:38:30.868 INFO:teuthology.orchestra.run.smithi098.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:38:30.868 INFO:teuthology.orchestra.run.smithi098.stdout:Group : Unspecified 2024-03-29T11:38:30.868 INFO:teuthology.orchestra.run.smithi098.stdout:Size : 8264254 2024-03-29T11:38:30.868 INFO:teuthology.orchestra.run.smithi098.stdout:License : GPLv3+ 2024-03-29T11:38:30.868 INFO:teuthology.orchestra.run.smithi098.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:38:30.868 INFO:teuthology.orchestra.run.smithi098.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:38:30.868 INFO:teuthology.orchestra.run.smithi098.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:38:30.868 INFO:teuthology.orchestra.run.smithi098.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:38:30.868 INFO:teuthology.orchestra.run.smithi098.stdout:Packager : builder@centos.org 2024-03-29T11:38:30.868 INFO:teuthology.orchestra.run.smithi098.stdout:Vendor : CentOS 2024-03-29T11:38:30.868 INFO:teuthology.orchestra.run.smithi098.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:38:30.868 INFO:teuthology.orchestra.run.smithi098.stdout:Summary : Support tools for GRUB. 2024-03-29T11:38:30.868 INFO:teuthology.orchestra.run.smithi098.stdout:Description : 2024-03-29T11:38:30.868 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:38:30.869 INFO:teuthology.orchestra.run.smithi098.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:38:30.869 INFO:teuthology.orchestra.run.smithi098.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:38:30.869 INFO:teuthology.orchestra.run.smithi098.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:38:30.869 INFO:teuthology.orchestra.run.smithi098.stdout:hardware devices. 2024-03-29T11:38:30.869 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:38:30.869 INFO:teuthology.orchestra.run.smithi098.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:38:30.870 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:38:30.870 INFO:teuthology.task.kernel:Updating grub on smithi098 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:38:30.870 DEBUG:teuthology.orchestra.run.smithi098:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:38:30.998 INFO:teuthology.orchestra.run.smithi121.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:38:31.026 INFO:teuthology.orchestra.run.smithi121.stderr:done 2024-03-29T11:38:31.029 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2024-03-29T11:38:31.044 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.1QuLTJnwT0 2024-03-29T11:38:31.044 DEBUG:teuthology.orchestra.run.smithi121:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.1QuLTJnwT0 2024-03-29T11:38:31.112 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.1QuLTJnwT0 2024-03-29T11:38:31.247 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.1QuLTJnwT0 is 6KB 2024-03-29T11:38:31.296 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.1QuLTJnwT0 2024-03-29T11:38:31.310 DEBUG:teuthology.orchestra.run.smithi121:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:38:31.375 INFO:teuthology.orchestra.run.smithi121.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:38:31.375 INFO:teuthology.orchestra.run.smithi121.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:38:31.375 INFO:teuthology.orchestra.run.smithi121.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:38:31.375 INFO:teuthology.orchestra.run.smithi121.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:38:31.376 DEBUG:teuthology.orchestra.run.smithi121:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:38:31.414 INFO:teuthology.orchestra.run.smithi098.stderr:Generating grub configuration file ... 2024-03-29T11:38:31.515 DEBUG:teuthology.orchestra.run.smithi121:> sudo shutdown -r now 2024-03-29T11:38:32.744 INFO:teuthology.orchestra.run.smithi098.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:38:32.778 INFO:teuthology.orchestra.run.smithi098.stderr:done 2024-03-29T11:38:32.780 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-03-29T11:38:32.794 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.jmItwuVJz1 2024-03-29T11:38:32.795 DEBUG:teuthology.orchestra.run.smithi098:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.jmItwuVJz1 2024-03-29T11:38:32.863 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.jmItwuVJz1 2024-03-29T11:38:32.996 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.jmItwuVJz1 is 6KB 2024-03-29T11:38:33.046 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.jmItwuVJz1 2024-03-29T11:38:33.060 DEBUG:teuthology.orchestra.run.smithi098:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:38:33.132 INFO:teuthology.orchestra.run.smithi098.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:38:33.132 INFO:teuthology.orchestra.run.smithi098.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:38:33.132 INFO:teuthology.orchestra.run.smithi098.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:38:33.132 INFO:teuthology.orchestra.run.smithi098.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:38:33.134 DEBUG:teuthology.orchestra.run.smithi098:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:38:33.291 DEBUG:teuthology.orchestra.run.smithi098:> sudo shutdown -r now 2024-03-29T11:39:01.546 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T11:39:01.546 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-03-29T11:39:01.547 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:39:03.295 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T11:39:03.295 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-29T11:39:03.296 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:39:19.967 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.121 2024-03-29T11:39:21.758 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-03-29T11:39:28.974 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-03-29T11:39:28.975 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:39:30.761 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-29T11:39:30.761 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:39:32.030 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.121 2024-03-29T11:39:33.822 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-03-29T11:39:44.033 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-03-29T11:39:44.034 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:39:45.825 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-29T11:39:45.826 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:40:16.012 DEBUG:teuthology.orchestra.run.smithi121:> true 2024-03-29T11:40:16.423 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-03-29T11:40:16.424 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:40:16.424 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2024-03-29T11:40:16.437 INFO:teuthology.orchestra.run.smithi121.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:40:16.437 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:40:16.438 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:40:16.438 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:40:17.438 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T11:40:17.439 DEBUG:teuthology.orchestra.run.smithi121:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:40:17.510 INFO:teuthology.orchestra.run.smithi121.stdout:ttyS1 2024-03-29T11:40:17.538 DEBUG:teuthology.parallel:result is None 2024-03-29T11:40:17.705 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-03-29T11:40:18.116 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-29T11:40:18.117 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:40:18.117 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-03-29T11:40:18.131 INFO:teuthology.orchestra.run.smithi098.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:40:18.131 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:40:18.131 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:40:18.131 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:40:19.132 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T11:40:19.132 DEBUG:teuthology.orchestra.run.smithi098:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:40:19.197 INFO:teuthology.orchestra.run.smithi098.stdout:ttyS1 2024-03-29T11:40:19.242 DEBUG:teuthology.parallel:result is None 2024-03-29T11:40:19.243 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T11:40:19.251 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T11:40:19.251 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:40:19.254 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:40:19.268 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T11:40:19.275 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T11:40:19.282 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T11:40:19.283 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:40:19.308 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:40:19.497 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T11:40:19.504 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T11:40:19.504 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:40:19.506 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:40:19.534 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:40:19.539 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:40:19.547 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:40:19.555 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:40:19.556 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T11:40:19.563 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T11:40:19.563 DEBUG:teuthology.orchestra.run.smithi098:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:40:19.598 DEBUG:teuthology.orchestra.run.smithi121:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:40:19.626 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T11:40:19.635 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T11:40:19.635 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:40:19.667 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:40:19.681 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:40:19.759 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:40:19.820 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:40:19.895 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:40:19.943 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T11:40:19.943 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:40:20.010 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:40:20.048 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:40:20.104 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:40:20.179 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:40:20.227 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-29T11:40:20.227 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:40:20.294 DEBUG:teuthology.orchestra.run.smithi098:> sudo service rsyslog restart 2024-03-29T11:40:20.296 DEBUG:teuthology.orchestra.run.smithi121:> sudo service rsyslog restart 2024-03-29T11:40:20.348 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:40:20.388 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:40:20.829 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T11:40:20.837 INFO:teuthology.task.internal:Starting timer... 2024-03-29T11:40:20.837 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T11:40:20.849 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T11:40:20.859 DEBUG:teuthology.orchestra.run.smithi098:> sudo service auditd rotate 2024-03-29T11:40:20.917 INFO:teuthology.orchestra.run.smithi098.stdout:Rotating logs: 2024-03-29T11:40:20.918 DEBUG:teuthology.orchestra.run.smithi121:> sudo service auditd rotate 2024-03-29T11:40:20.975 INFO:teuthology.orchestra.run.smithi121.stdout:Rotating logs: 2024-03-29T11:40:20.976 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T11:40:20.976 DEBUG:teuthology.orchestra.run.smithi098:> /usr/sbin/getenforce 2024-03-29T11:40:20.996 INFO:teuthology.orchestra.run.smithi098.stdout:Permissive 2024-03-29T11:40:20.996 DEBUG:teuthology.orchestra.run.smithi121:> /usr/sbin/getenforce 2024-03-29T11:40:21.029 INFO:teuthology.orchestra.run.smithi121.stdout:Permissive 2024-03-29T11:40:21.030 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi098.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi121.front.sepia.ceph.com': 'permissive'} 2024-03-29T11:40:21.030 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:40:21.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:40:21.063 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:40:21.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:40:21.097 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T11:40:21.098 DEBUG:teuthology.orchestra.run.smithi098:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:40:21.140 DEBUG:teuthology.orchestra.run.smithi121:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:40:21.171 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T11:40:21.182 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T11:40:21.200 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T11:40:21.201 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi098.front.sepia.ceph.com,smithi121.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T11:44:35.076 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi098.front.sepia.ceph.com'), Remote(name='ubuntu@smithi121.front.sepia.ceph.com')] 2024-03-29T11:44:35.078 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-29T11:44:35.079 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:44:35.173 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-03-29T11:44:35.257 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-29T11:44:35.257 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-03-29T11:44:35.258 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:44:35.363 DEBUG:teuthology.orchestra.run.smithi121:> true 2024-03-29T11:44:35.441 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-03-29T11:44:35.442 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T11:44:35.458 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T11:44:35.459 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:44:35.459 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:44:35.464 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:44:35.464 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:44:35.494 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:44:35.511 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:44:35.515 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:44:35.532 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:44:35.541 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: ntpd: command not found 2024-03-29T11:44:35.554 INFO:teuthology.orchestra.run.smithi098.stdout:506 Cannot talk to daemon 2024-03-29T11:44:35.561 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: ntpd: command not found 2024-03-29T11:44:35.571 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:44:35.575 INFO:teuthology.orchestra.run.smithi121.stdout:506 Cannot talk to daemon 2024-03-29T11:44:35.587 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:44:35.591 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:44:35.607 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:44:35.646 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:44:35.649 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:44:35.649 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-03-29T11:44:35.649 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:44:35.649 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:44:35.649 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:44:35.649 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:44:35.658 INFO:teuthology.orchestra.run.smithi121.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:44:35.662 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:44:35.662 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2024-03-29T11:44:35.662 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:44:35.662 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:44:35.662 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:44:35.662 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:44:35.662 INFO:teuthology.run_tasks:Running task install... 2024-03-29T11:44:35.679 DEBUG:teuthology.task.install:project ceph 2024-03-29T11:44:35.679 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T11:44:35.679 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T11:44:35.679 INFO:teuthology.task.install:Using flavor: default 2024-03-29T11:44:35.686 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T11:44:35.686 INFO:teuthology.task.install:extra packages: [] 2024-03-29T11:44:35.687 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:44:35.688 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:44:35.690 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:44:35.691 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:44:35.845 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:44:35.845 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:44:35.926 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:44:35.927 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:44:36.213 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:44:36.214 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-29T11:44:36.214 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:44:36.243 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:44:36.243 DEBUG:teuthology.orchestra.run.smithi121:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:44:36.287 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:44:36.287 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T11:44:36.287 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:44:36.312 DEBUG:teuthology.orchestra.run.smithi121:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:44:36.317 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:44:36.317 DEBUG:teuthology.orchestra.run.smithi098:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:44:36.386 DEBUG:teuthology.orchestra.run.smithi098:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:44:36.393 DEBUG:teuthology.orchestra.run.smithi121:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:44:36.459 INFO:teuthology.orchestra.run.smithi121.stdout:check_obsoletes = 1 2024-03-29T11:44:36.460 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum clean all 2024-03-29T11:44:36.466 DEBUG:teuthology.orchestra.run.smithi098:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:44:36.533 INFO:teuthology.orchestra.run.smithi098.stdout:check_obsoletes = 1 2024-03-29T11:44:36.535 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum clean all 2024-03-29T11:44:36.714 INFO:teuthology.orchestra.run.smithi121.stdout:74 files removed 2024-03-29T11:44:36.742 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-radosgw 2024-03-29T11:44:36.798 INFO:teuthology.orchestra.run.smithi098.stdout:74 files removed 2024-03-29T11:44:36.827 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-radosgw 2024-03-29T11:44:37.890 INFO:teuthology.orchestra.run.smithi098.stdout:ceph packages for x86_64 107 kB/s | 82 kB 00:00 2024-03-29T11:44:39.405 INFO:teuthology.orchestra.run.smithi098.stdout:ceph noarch packages 13 kB/s | 10 kB 00:00 2024-03-29T11:44:39.405 INFO:teuthology.orchestra.run.smithi121.stdout:ceph packages for x86_64 80 kB/s | 82 kB 00:01 2024-03-29T11:44:39.405 INFO:teuthology.orchestra.run.smithi121.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-29T11:44:39.674 INFO:teuthology.orchestra.run.smithi098.stdout:ceph source packages 2.0 kB/s | 1.9 kB 00:00 2024-03-29T11:44:39.838 INFO:teuthology.orchestra.run.smithi121.stdout:ceph source packages 2.2 kB/s | 1.9 kB 00:00 2024-03-29T11:44:40.084 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-29T11:44:40.506 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-29T11:44:41.768 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-9 - BaseOS 3.4 MB/s | 2.0 MB 00:00 2024-03-29T11:44:42.067 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS-9 - BaseOS 4.4 MB/s | 2.0 MB 00:00 2024-03-29T11:44:43.116 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-29T11:44:45.167 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:00 2024-03-29T11:44:49.268 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - CRB 9.9 MB/s | 6.1 MB 00:00 2024-03-29T11:44:50.718 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - Extras packages 51 kB/s | 16 kB 00:00 2024-03-29T11:44:50.908 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 934 kB/s | 8.0 MB 00:08 2024-03-29T11:44:52.330 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-29T11:44:57.294 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-03-29T11:45:00.033 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:45:00.047 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:45:00.047 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-03-29T11:45:00.047 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:45:00.047 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-29T11:45:00.047 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:45:00.047 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrading: 2024-03-29T11:45:00.047 INFO:teuthology.orchestra.run.smithi121.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:45:00.048 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:45:00.048 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-03-29T11:45:00.048 INFO:teuthology.orchestra.run.smithi121.stdout: boost-program-options 2024-03-29T11:45:00.048 INFO:teuthology.orchestra.run.smithi121.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:45:00.048 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:45:00.048 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:45:00.048 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:45:00.048 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:45:00.048 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:45:00.048 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:45:00.048 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:45:00.048 INFO:teuthology.orchestra.run.smithi121.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:45:00.048 INFO:teuthology.orchestra.run.smithi121.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:45:00.048 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:45:00.048 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:45:00.049 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:45:00.049 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:45:00.049 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:45:00.049 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:45:00.049 INFO:teuthology.orchestra.run.smithi121.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:45:00.049 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse 2024-03-29T11:45:00.049 INFO:teuthology.orchestra.run.smithi121.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:45:00.049 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:45:00.049 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:45:00.049 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:45:00.049 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:45:00.049 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:45:00.049 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:45:00.049 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:45:00.050 INFO:teuthology.orchestra.run.smithi121.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:45:00.050 INFO:teuthology.orchestra.run.smithi121.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:45:00.050 INFO:teuthology.orchestra.run.smithi121.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:45:00.050 INFO:teuthology.orchestra.run.smithi121.stdout:Installing weak dependencies: 2024-03-29T11:45:00.050 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:45:00.050 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:45:00.050 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-29T11:45:00.050 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:45:00.050 INFO:teuthology.orchestra.run.smithi121.stdout:Install 30 Packages 2024-03-29T11:45:00.050 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrade 2 Packages 2024-03-29T11:45:00.050 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:45:00.051 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 58 M 2024-03-29T11:45:00.051 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-29T11:45:02.712 INFO:teuthology.orchestra.run.smithi121.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 2.1 MB/s | 5.3 MB 00:02 2024-03-29T11:45:02.762 INFO:teuthology.orchestra.run.smithi121.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 506 kB/s | 25 kB 00:00 2024-03-29T11:45:02.954 INFO:teuthology.orchestra.run.smithi121.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 3.8 MB/s | 742 kB 00:00 2024-03-29T11:45:03.087 INFO:teuthology.orchestra.run.smithi121.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 3.7 MB/s | 498 kB 00:00 2024-03-29T11:45:03.912 INFO:teuthology.orchestra.run.smithi121.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 2.6 MB/s | 9.7 MB 00:03 2024-03-29T11:45:03.954 INFO:teuthology.orchestra.run.smithi121.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 1.1 MB/s | 45 kB 00:00 2024-03-29T11:45:04.021 INFO:teuthology.orchestra.run.smithi121.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.9 MB/s | 131 kB 00:00 2024-03-29T11:45:04.096 INFO:teuthology.orchestra.run.smithi121.stdout:(8/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 5.1 MB/s | 5.1 MB 00:01 2024-03-29T11:45:04.138 INFO:teuthology.orchestra.run.smithi121.stdout:(9/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 1.4 MB/s | 162 kB 00:00 2024-03-29T11:45:04.189 INFO:teuthology.orchestra.run.smithi121.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 3.4 MB/s | 320 kB 00:00 2024-03-29T11:45:04.239 INFO:teuthology.orchestra.run.smithi121.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 2.9 MB/s | 298 kB 00:00 2024-03-29T11:45:04.264 INFO:teuthology.orchestra.run.smithi121.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.3 MB/s | 99 kB 00:00 2024-03-29T11:45:04.381 INFO:teuthology.orchestra.run.smithi121.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 747 kB/s | 106 kB 00:00 2024-03-29T11:45:04.406 INFO:teuthology.orchestra.run.smithi121.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 159 kB 00:00 2024-03-29T11:45:04.431 INFO:teuthology.orchestra.run.smithi121.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 883 kB/s | 44 kB 00:00 2024-03-29T11:45:04.490 INFO:teuthology.orchestra.run.smithi121.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 661 kB 00:00 2024-03-29T11:45:04.515 INFO:teuthology.orchestra.run.smithi121.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2024-03-29T11:45:04.541 INFO:teuthology.orchestra.run.smithi121.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 829 kB/s | 41 kB 00:00 2024-03-29T11:45:04.566 INFO:teuthology.orchestra.run.smithi121.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-03-29T11:45:04.583 INFO:teuthology.orchestra.run.smithi121.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 835 kB/s | 35 kB 00:00 2024-03-29T11:45:04.600 INFO:teuthology.orchestra.run.smithi121.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-29T11:45:05.134 INFO:teuthology.orchestra.run.smithi121.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 561 kB/s | 309 kB 00:00 2024-03-29T11:45:05.384 INFO:teuthology.orchestra.run.smithi121.stdout:(23/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 3.8 MB/s | 20 MB 00:05 2024-03-29T11:45:05.401 INFO:teuthology.orchestra.run.smithi121.stdout:(24/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 97 kB/s | 26 kB 00:00 2024-03-29T11:45:05.509 INFO:teuthology.orchestra.run.smithi121.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 622 kB/s | 67 kB 00:00 2024-03-29T11:45:05.584 INFO:teuthology.orchestra.run.smithi121.stdout:(26/32): libarrow-9.0.0-10.el9.x86_64.rpm 4.5 MB/s | 4.4 MB 00:00 2024-03-29T11:45:05.685 INFO:teuthology.orchestra.run.smithi121.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-03-29T11:45:05.710 INFO:teuthology.orchestra.run.smithi121.stdout:(28/32): liboath-2.6.7-2.el9.x86_64.rpm 149 kB/s | 49 kB 00:00 2024-03-29T11:45:05.761 INFO:teuthology.orchestra.run.smithi121.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.3 MB/s | 838 kB 00:00 2024-03-29T11:45:05.861 INFO:teuthology.orchestra.run.smithi121.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 9.0 MB/s | 1.6 MB 00:00 2024-03-29T11:45:06.428 INFO:teuthology.orchestra.run.smithi121.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 4.7 MB/s | 3.1 MB 00:00 2024-03-29T11:45:06.603 INFO:teuthology.orchestra.run.smithi121.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 3.8 MB/s | 3.4 MB 00:00 2024-03-29T11:45:06.607 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:45:06.607 INFO:teuthology.orchestra.run.smithi121.stdout:Total 8.8 MB/s | 58 MB 00:06 2024-03-29T11:45:06.608 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-29T11:45:06.645 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-29T11:45:06.645 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-29T11:45:07.114 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-29T11:45:07.115 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-29T11:45:08.601 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-29T11:45:08.688 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:45:08.724 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:45:08.749 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:45:08.772 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:45:08.790 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:45:08.956 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:45:09.010 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:45:09.337 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:45:09.401 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:45:09.484 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:45:09.574 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:45:09.603 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:45:09.665 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:45:09.694 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:45:09.738 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:45:09.807 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:45:09.830 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:45:09.885 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:45:09.920 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:45:10.031 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:45:10.063 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:45:10.103 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:45:10.136 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:45:10.164 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:45:10.197 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:45:10.517 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:45:10.544 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:45:10.616 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:45:10.649 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:45:10.692 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:45:11.146 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:45:11.171 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:45:11.225 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:45:11.294 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:45:11.323 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:45:13.124 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 916 kB/s | 19 MB 00:21 2024-03-29T11:45:13.505 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:45:13.537 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:45:14.092 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:45:14.140 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:45:14.235 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:45:14.236 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:45:14.236 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:45:14.237 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:45:14.274 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:45:17.286 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 9.1 MB/s | 6.1 MB 00:00 2024-03-29T11:45:18.654 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 83 kB/s | 16 kB 00:00 2024-03-29T11:45:20.013 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-29T11:45:23.816 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:45:23.816 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /sys 2024-03-29T11:45:23.816 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /proc 2024-03-29T11:45:23.816 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /mnt 2024-03-29T11:45:23.816 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /var/tmp 2024-03-29T11:45:23.816 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /home 2024-03-29T11:45:23.817 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /root 2024-03-29T11:45:23.817 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /tmp 2024-03-29T11:45:23.817 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:45:24.728 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:45:24.752 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:45:24.753 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:45:24.753 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:45:24.753 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:45:24.753 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:45:24.753 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:45:24.754 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:45:24.831 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:45:24.831 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:45:25.045 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-29T11:45:25.996 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:45:25.996 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:45:25.996 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:45:25.996 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:45:25.996 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:45:25.996 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:45:25.997 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:45:25.997 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:45:25.997 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:45:25.997 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:45:25.997 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:45:25.997 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:45:25.997 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:45:25.997 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:45:25.997 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:45:25.997 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:45:25.997 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:45:25.998 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:45:25.998 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:45:25.998 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:45:25.998 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:45:25.998 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:45:25.998 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:45:25.998 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:45:25.998 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:45:25.998 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:45:25.998 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:45:25.998 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:45:25.999 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:45:25.999 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:45:25.999 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:45:25.999 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:45:25.999 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:45:25.999 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:45:27.064 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:45:27.065 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:45:27.065 INFO:teuthology.orchestra.run.smithi121.stdout:Upgraded: 2024-03-29T11:45:27.065 INFO:teuthology.orchestra.run.smithi121.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:45:27.065 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:45:27.065 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-29T11:45:27.065 INFO:teuthology.orchestra.run.smithi121.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:45:27.065 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:45:27.065 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:45:27.065 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:45:27.065 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:45:27.065 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:45:27.065 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:45:27.065 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:45:27.065 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:45:27.066 INFO:teuthology.orchestra.run.smithi121.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:45:27.066 INFO:teuthology.orchestra.run.smithi121.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:45:27.066 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:45:27.066 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:45:27.066 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:45:27.066 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:45:27.066 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:45:27.066 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:45:27.066 INFO:teuthology.orchestra.run.smithi121.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:45:27.066 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:45:27.066 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:45:27.066 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:45:27.066 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:45:27.066 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:45:27.067 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:45:27.067 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:45:27.067 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:45:27.067 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:45:27.067 INFO:teuthology.orchestra.run.smithi121.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:45:27.067 INFO:teuthology.orchestra.run.smithi121.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:45:27.067 INFO:teuthology.orchestra.run.smithi121.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:45:27.067 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:45:27.067 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:45:27.357 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-test 2024-03-29T11:45:27.660 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T11:45:27.667 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T11:45:27.667 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-29T11:45:27.667 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T11:45:27.667 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-29T11:45:27.667 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:45:27.667 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrading: 2024-03-29T11:45:27.667 INFO:teuthology.orchestra.run.smithi098.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:45:27.667 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:45:27.667 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-29T11:45:27.668 INFO:teuthology.orchestra.run.smithi098.stdout: boost-program-options 2024-03-29T11:45:27.668 INFO:teuthology.orchestra.run.smithi098.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:45:27.668 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:45:27.668 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:45:27.668 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:45:27.668 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:45:27.668 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:45:27.668 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:45:27.668 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:45:27.668 INFO:teuthology.orchestra.run.smithi098.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:45:27.668 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:45:27.668 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:45:27.668 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:45:27.669 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:45:27.669 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:45:27.669 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:45:27.669 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:45:27.669 INFO:teuthology.orchestra.run.smithi098.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:45:27.669 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse 2024-03-29T11:45:27.669 INFO:teuthology.orchestra.run.smithi098.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:45:27.669 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:45:27.669 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:45:27.669 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:45:27.669 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:45:27.669 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:45:27.669 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:45:27.669 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:45:27.669 INFO:teuthology.orchestra.run.smithi098.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:45:27.670 INFO:teuthology.orchestra.run.smithi098.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:45:27.670 INFO:teuthology.orchestra.run.smithi098.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:45:27.670 INFO:teuthology.orchestra.run.smithi098.stdout:Installing weak dependencies: 2024-03-29T11:45:27.670 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:45:27.670 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:45:27.670 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-29T11:45:27.670 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T11:45:27.670 INFO:teuthology.orchestra.run.smithi098.stdout:Install 30 Packages 2024-03-29T11:45:27.670 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrade 2 Packages 2024-03-29T11:45:27.670 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:45:27.671 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 58 M 2024-03-29T11:45:27.671 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-29T11:45:27.906 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:30 ago on Fri 29 Mar 2024 11:44:57 AM UTC. 2024-03-29T11:45:28.054 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:45:28.055 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:45:28.055 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-03-29T11:45:28.055 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:45:28.055 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-29T11:45:28.055 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:45:28.055 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-03-29T11:45:28.055 INFO:teuthology.orchestra.run.smithi121.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:45:28.055 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:45:28.055 INFO:teuthology.orchestra.run.smithi121.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:45:28.056 INFO:teuthology.orchestra.run.smithi121.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:45:28.056 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:45:28.056 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:45:28.056 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-29T11:45:28.056 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:45:28.056 INFO:teuthology.orchestra.run.smithi121.stdout:Install 6 Packages 2024-03-29T11:45:28.056 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:45:28.056 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 48 M 2024-03-29T11:45:28.057 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 204 M 2024-03-29T11:45:28.057 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-29T11:45:28.478 INFO:teuthology.orchestra.run.smithi121.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-03-29T11:45:28.528 INFO:teuthology.orchestra.run.smithi121.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.8 MB/s | 300 kB 00:00 2024-03-29T11:45:28.553 INFO:teuthology.orchestra.run.smithi121.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.5 MB/s | 63 kB 00:00 2024-03-29T11:45:28.653 INFO:teuthology.orchestra.run.smithi121.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.8 MB/s | 187 kB 00:00 2024-03-29T11:45:28.895 INFO:teuthology.orchestra.run.smithi121.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 289 kB/s | 163 kB 00:00 2024-03-29T11:45:31.029 INFO:teuthology.orchestra.run.smithi098.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 1.7 MB/s | 5.3 MB 00:03 2024-03-29T11:45:31.180 INFO:teuthology.orchestra.run.smithi098.stdout:(2/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 2.9 MB/s | 9.7 MB 00:03 2024-03-29T11:45:31.196 INFO:teuthology.orchestra.run.smithi098.stdout:(3/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 151 kB/s | 25 kB 00:00 2024-03-29T11:45:31.280 INFO:teuthology.orchestra.run.smithi098.stdout:(4/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 7.2 MB/s | 742 kB 00:00 2024-03-29T11:45:31.405 INFO:teuthology.orchestra.run.smithi098.stdout:(5/32): libradosstriper1-19.0.0-1439.g5dcb190d. 2.3 MB/s | 498 kB 00:00 2024-03-29T11:45:31.463 INFO:teuthology.orchestra.run.smithi098.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 774 kB/s | 45 kB 00:00 2024-03-29T11:45:31.495 INFO:teuthology.orchestra.run.smithi121.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 15 MB/s | 47 MB 00:03 2024-03-29T11:45:31.498 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:45:31.498 INFO:teuthology.orchestra.run.smithi121.stdout:Total 14 MB/s | 48 MB 00:03 2024-03-29T11:45:31.512 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-29T11:45:31.533 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-29T11:45:31.533 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-29T11:45:31.547 INFO:teuthology.orchestra.run.smithi098.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.5 MB/s | 131 kB 00:00 2024-03-29T11:45:31.622 INFO:teuthology.orchestra.run.smithi098.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 2.1 MB/s | 162 kB 00:00 2024-03-29T11:45:31.756 INFO:teuthology.orchestra.run.smithi098.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 2.3 MB/s | 320 kB 00:00 2024-03-29T11:45:31.912 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-29T11:45:31.912 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-29T11:45:31.914 INFO:teuthology.orchestra.run.smithi098.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 1.8 MB/s | 298 kB 00:00 2024-03-29T11:45:32.015 INFO:teuthology.orchestra.run.smithi098.stdout:(11/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 6.9 MB/s | 5.1 MB 00:00 2024-03-29T11:45:32.057 INFO:teuthology.orchestra.run.smithi098.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 699 kB/s | 99 kB 00:00 2024-03-29T11:45:32.291 INFO:teuthology.orchestra.run.smithi098.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 384 kB/s | 106 kB 00:00 2024-03-29T11:45:32.316 INFO:teuthology.orchestra.run.smithi098.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 681 kB/s | 159 kB 00:00 2024-03-29T11:45:32.341 INFO:teuthology.orchestra.run.smithi098.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 882 kB/s | 44 kB 00:00 2024-03-29T11:45:32.400 INFO:teuthology.orchestra.run.smithi098.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.8 MB/s | 661 kB 00:00 2024-03-29T11:45:32.425 INFO:teuthology.orchestra.run.smithi098.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2024-03-29T11:45:32.450 INFO:teuthology.orchestra.run.smithi098.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 827 kB/s | 41 kB 00:00 2024-03-29T11:45:32.476 INFO:teuthology.orchestra.run.smithi098.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-03-29T11:45:32.501 INFO:teuthology.orchestra.run.smithi098.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 697 kB/s | 35 kB 00:00 2024-03-29T11:45:32.503 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-29T11:45:32.526 INFO:teuthology.orchestra.run.smithi098.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 923 kB/s | 46 kB 00:00 2024-03-29T11:45:32.579 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:45:32.625 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:45:32.665 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:45:32.712 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:45:32.740 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:45:33.479 INFO:teuthology.orchestra.run.smithi098.stdout:(22/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 3.5 MB/s | 20 MB 00:05 2024-03-29T11:45:33.602 INFO:teuthology.orchestra.run.smithi098.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 208 kB/s | 26 kB 00:00 2024-03-29T11:45:33.710 INFO:teuthology.orchestra.run.smithi098.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 447 kB/s | 49 kB 00:00 2024-03-29T11:45:33.770 INFO:teuthology.orchestra.run.smithi098.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-29T11:45:33.920 INFO:teuthology.orchestra.run.smithi098.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.5 MB/s | 838 kB 00:00 2024-03-29T11:45:33.987 INFO:teuthology.orchestra.run.smithi098.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-03-29T11:45:34.129 INFO:teuthology.orchestra.run.smithi098.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-03-29T11:45:34.947 INFO:teuthology.orchestra.run.smithi098.stdout:(29/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 4.1 MB/s | 3.4 MB 00:00 2024-03-29T11:45:35.712 INFO:teuthology.orchestra.run.smithi098.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 4.1 MB/s | 3.1 MB 00:00 2024-03-29T11:45:37.197 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:45:37.280 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:45:39.583 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:45:39.583 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:45:39.583 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:45:39.584 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:45:39.584 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:45:39.584 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:45:39.927 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:45:39.928 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:45:39.928 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-29T11:45:39.928 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:45:39.928 INFO:teuthology.orchestra.run.smithi121.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:45:39.928 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:45:39.928 INFO:teuthology.orchestra.run.smithi121.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:45:39.928 INFO:teuthology.orchestra.run.smithi121.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:45:39.928 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:45:39.928 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:45:39.928 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:45:40.209 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph 2024-03-29T11:45:40.646 INFO:teuthology.orchestra.run.smithi098.stdout:(31/32): libarrow-9.0.0-10.el9.x86_64.rpm 557 kB/s | 4.4 MB 00:08 2024-03-29T11:45:40.797 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:43 ago on Fri 29 Mar 2024 11:44:57 AM UTC. 2024-03-29T11:45:40.957 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:45:40.960 INFO:teuthology.orchestra.run.smithi121.stdout:========================================================================================= 2024-03-29T11:45:40.960 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-03-29T11:45:40.960 INFO:teuthology.orchestra.run.smithi121.stdout:========================================================================================= 2024-03-29T11:45:40.960 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-29T11:45:40.960 INFO:teuthology.orchestra.run.smithi121.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:45:40.960 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-03-29T11:45:40.960 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:45:40.960 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:45:40.960 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:45:40.961 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:45:40.961 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:45:40.961 INFO:teuthology.orchestra.run.smithi121.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:45:40.961 INFO:teuthology.orchestra.run.smithi121.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:45:40.961 INFO:teuthology.orchestra.run.smithi121.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:45:40.961 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:45:40.961 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:45:40.961 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:45:40.961 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:45:40.961 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:45:40.961 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:45:40.961 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:45:40.961 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:45:40.961 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:45:40.962 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:45:40.962 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:45:40.962 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:45:40.962 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:45:40.962 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:45:40.962 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:45:40.962 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:45:40.962 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:45:40.963 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:45:40.963 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:45:40.963 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:45:40.963 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:45:40.963 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:45:40.963 INFO:teuthology.orchestra.run.smithi121.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:45:40.963 INFO:teuthology.orchestra.run.smithi121.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:45:40.963 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:45:40.963 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:45:40.963 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:45:40.963 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:45:40.963 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-29T11:45:40.963 INFO:teuthology.orchestra.run.smithi121.stdout:========================================================================================= 2024-03-29T11:45:40.964 INFO:teuthology.orchestra.run.smithi121.stdout:Install 36 Packages 2024-03-29T11:45:40.964 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:45:40.964 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 30 M 2024-03-29T11:45:40.964 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 107 M 2024-03-29T11:45:40.964 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-29T11:45:41.725 INFO:teuthology.orchestra.run.smithi121.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 29 kB/s | 6.6 kB 00:00 2024-03-29T11:45:41.842 INFO:teuthology.orchestra.run.smithi121.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 4.3 MB/s | 1.5 MB 00:00 2024-03-29T11:45:42.684 INFO:teuthology.orchestra.run.smithi121.stdout:(3/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86_ 20 MB/s | 17 MB 00:00 2024-03-29T11:45:42.776 INFO:teuthology.orchestra.run.smithi121.stdout:(4/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 1.7 MB/s | 2.1 MB 00:01 2024-03-29T11:45:42.801 INFO:teuthology.orchestra.run.smithi121.stdout:(5/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 2.1 MB/s | 247 kB 00:00 2024-03-29T11:45:42.935 INFO:teuthology.orchestra.run.smithi121.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 144 kB 00:00 2024-03-29T11:45:42.961 INFO:teuthology.orchestra.run.smithi121.stdout:(7/36): lua-5.4.4-4.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-03-29T11:45:42.977 INFO:teuthology.orchestra.run.smithi121.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 756 kB/s | 32 kB 00:00 2024-03-29T11:45:42.994 INFO:teuthology.orchestra.run.smithi121.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T11:45:43.011 INFO:teuthology.orchestra.run.smithi121.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-29T11:45:43.036 INFO:teuthology.orchestra.run.smithi121.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-29T11:45:43.095 INFO:teuthology.orchestra.run.smithi121.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-29T11:45:43.120 INFO:teuthology.orchestra.run.smithi121.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-29T11:45:43.145 INFO:teuthology.orchestra.run.smithi121.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-29T11:45:43.171 INFO:teuthology.orchestra.run.smithi121.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-03-29T11:45:43.330 INFO:teuthology.orchestra.run.smithi121.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 141 kB/s | 22 kB 00:00 2024-03-29T11:45:43.346 INFO:teuthology.orchestra.run.smithi121.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 622 kB/s | 125 kB 00:00 2024-03-29T11:45:43.488 INFO:teuthology.orchestra.run.smithi121.stdout:(18/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 2.7 MB/s | 4.7 MB 00:01 2024-03-29T11:45:43.605 INFO:teuthology.orchestra.run.smithi121.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 548 kB/s | 151 kB 00:00 2024-03-29T11:45:43.731 INFO:teuthology.orchestra.run.smithi121.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.8 MB/s | 358 kB 00:00 2024-03-29T11:45:43.790 INFO:teuthology.orchestra.run.smithi121.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 183 kB/s | 11 kB 00:00 2024-03-29T11:45:43.873 INFO:teuthology.orchestra.run.smithi121.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 212 kB/s | 18 kB 00:00 2024-03-29T11:45:43.957 INFO:teuthology.orchestra.run.smithi121.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 277 kB/s | 23 kB 00:00 2024-03-29T11:45:44.057 INFO:teuthology.orchestra.run.smithi121.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 195 kB/s | 19 kB 00:00 2024-03-29T11:45:44.157 INFO:teuthology.orchestra.run.smithi121.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 196 kB/s | 20 kB 00:00 2024-03-29T11:45:44.257 INFO:teuthology.orchestra.run.smithi121.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 463 kB/s | 46 kB 00:00 2024-03-29T11:45:44.357 INFO:teuthology.orchestra.run.smithi121.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 789 kB/s | 79 kB 00:00 2024-03-29T11:45:44.483 INFO:teuthology.orchestra.run.smithi121.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-03-29T11:45:44.541 INFO:teuthology.orchestra.run.smithi121.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 281 kB/s | 16 kB 00:00 2024-03-29T11:45:44.600 INFO:teuthology.orchestra.run.smithi121.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-03-29T11:45:44.708 INFO:teuthology.orchestra.run.smithi121.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 330 kB/s | 36 kB 00:00 2024-03-29T11:45:44.775 INFO:teuthology.orchestra.run.smithi121.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-03-29T11:45:44.851 INFO:teuthology.orchestra.run.smithi121.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.6 MB/s | 427 kB 00:00 2024-03-29T11:45:44.909 INFO:teuthology.orchestra.run.smithi121.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 342 kB/s | 20 kB 00:00 2024-03-29T11:45:46.717 INFO:teuthology.orchestra.run.smithi121.stdout:(35/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 53 kB/s | 172 kB 00:03 2024-03-29T11:45:48.512 INFO:teuthology.orchestra.run.smithi098.stdout:(32/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 19 kB/s | 309 kB 00:16 2024-03-29T11:45:48.517 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:45:48.517 INFO:teuthology.orchestra.run.smithi098.stdout:Total 2.8 MB/s | 58 MB 00:20 2024-03-29T11:45:48.517 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-29T11:45:48.554 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-29T11:45:48.554 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-29T11:45:49.038 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-29T11:45:49.038 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-29T11:45:50.490 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-29T11:45:50.574 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:45:50.608 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:45:50.634 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:45:50.657 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:45:50.673 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:45:50.832 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:45:50.904 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:45:51.229 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:45:51.294 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:45:51.385 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:45:51.472 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:45:51.504 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:45:51.566 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:45:51.595 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:45:51.639 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:45:51.706 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:45:51.731 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:45:51.778 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:45:51.813 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:45:51.920 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:45:51.965 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:45:52.006 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:45:52.046 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:45:52.075 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:45:52.107 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:45:52.417 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:45:52.446 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:45:52.536 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:45:52.567 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:45:52.606 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:45:53.057 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:45:53.082 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:45:53.137 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:45:53.205 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:45:53.243 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:45:55.545 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:45:55.581 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:45:56.150 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:45:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:45:56.367 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:45:56.367 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:45:56.367 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:45:56.367 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:45:56.403 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:45:58.666 INFO:teuthology.orchestra.run.smithi121.stdout:(36/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 2.8 kB/s | 43 kB 00:15 2024-03-29T11:45:58.670 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:45:58.670 INFO:teuthology.orchestra.run.smithi121.stdout:Total 1.7 MB/s | 30 MB 00:17 2024-03-29T11:45:58.695 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-29T11:45:58.737 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-29T11:45:58.737 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-29T11:45:59.154 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-29T11:45:59.154 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-29T11:46:00.563 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-29T11:46:00.618 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:46:00.651 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:46:00.679 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:46:00.749 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:46:00.794 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:46:00.827 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:46:00.917 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:46:00.996 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:46:01.038 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:46:01.109 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:46:01.163 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:46:01.202 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:46:01.251 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:46:01.306 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:46:01.498 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:46:01.542 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:46:01.571 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:46:01.605 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:46:01.635 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:46:01.711 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:46:01.754 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:46:01.787 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:46:01.827 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:46:01.862 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:46:01.897 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:46:01.986 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:46:02.019 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:46:02.091 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:46:02.227 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:46:02.360 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:46:02.520 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:46:02.550 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:46:02.576 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:46:02.576 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:46:02.576 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:46:02.576 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:46:02.576 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:46:02.576 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:04.001 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:46:04.027 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:46:04.028 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:46:04.028 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:46:04.028 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:46:04.028 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:46:04.028 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:04.468 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:46:04.491 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:46:04.491 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:46:04.491 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:46:04.491 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:46:04.491 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:46:04.492 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:04.694 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:46:04.730 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:46:04.731 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:46:04.731 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:46:04.731 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:46:04.731 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:46:04.731 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:04.782 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:46:05.661 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:46:05.661 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /sys 2024-03-29T11:46:05.661 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /proc 2024-03-29T11:46:05.662 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /mnt 2024-03-29T11:46:05.662 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /var/tmp 2024-03-29T11:46:05.662 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /home 2024-03-29T11:46:05.662 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /root 2024-03-29T11:46:05.662 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /tmp 2024-03-29T11:46:05.662 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:46:06.329 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:46:06.329 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:46:06.329 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:46:06.329 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:46:06.329 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:46:06.330 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:46:06.330 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:46:06.330 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:46:06.330 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:46:06.330 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:46:06.330 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:46:06.330 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:46:06.330 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:46:06.330 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:46:06.330 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:46:06.330 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:46:06.330 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:46:06.330 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:46:06.331 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:46:06.331 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:46:06.331 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:46:06.331 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:46:06.331 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:46:06.331 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:46:06.331 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:46:06.331 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:46:06.331 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:46:06.331 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:46:06.332 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:46:06.332 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:46:06.332 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:46:06.332 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:46:06.332 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:46:06.333 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:46:06.333 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:46:06.333 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:46:06.584 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:46:06.609 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:46:06.609 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:46:06.609 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:46:06.609 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:46:06.609 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:46:06.609 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:46:06.611 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:46:06.679 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:46:06.679 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:46:07.221 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:46:07.221 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:07.221 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-29T11:46:07.221 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:46:07.221 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:46:07.221 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:46:07.221 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:46:07.222 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:46:07.222 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:46:07.222 INFO:teuthology.orchestra.run.smithi121.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:46:07.222 INFO:teuthology.orchestra.run.smithi121.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:46:07.222 INFO:teuthology.orchestra.run.smithi121.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:46:07.222 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:46:07.222 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:46:07.222 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:46:07.222 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:46:07.222 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:46:07.222 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:46:07.222 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:46:07.222 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:46:07.222 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:46:07.223 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:46:07.223 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:46:07.223 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:46:07.223 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:46:07.223 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:46:07.223 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:46:07.223 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:46:07.223 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:46:07.223 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:46:07.223 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:46:07.223 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:46:07.223 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:46:07.223 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:46:07.223 INFO:teuthology.orchestra.run.smithi121.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:46:07.223 INFO:teuthology.orchestra.run.smithi121.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:46:07.223 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:46:07.224 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:46:07.224 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:46:07.224 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:07.224 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:46:07.500 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-base 2024-03-29T11:46:07.825 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:46:07.825 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:46:07.826 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:46:07.826 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:46:07.826 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:46:07.826 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:46:07.826 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:46:07.826 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:46:07.826 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:46:07.826 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:46:07.826 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:46:07.826 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:46:07.826 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:46:07.826 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:46:07.826 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:46:07.826 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:46:07.827 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:46:07.827 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:46:07.827 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:46:07.827 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:46:07.827 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:46:07.827 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:46:07.827 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:46:07.827 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:46:07.827 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:46:07.827 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:46:07.827 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:46:07.827 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:46:07.827 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:46:07.827 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:46:07.828 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:46:07.828 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:46:07.828 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:46:07.828 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:46:08.060 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:11 ago on Fri 29 Mar 2024 11:44:57 AM UTC. 2024-03-29T11:46:08.157 INFO:teuthology.orchestra.run.smithi121.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:46:08.209 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:46:08.210 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-03-29T11:46:08.210 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:46:08.273 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install cephadm 2024-03-29T11:46:08.708 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:46:08.708 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:46:08.708 INFO:teuthology.orchestra.run.smithi098.stdout:Upgraded: 2024-03-29T11:46:08.708 INFO:teuthology.orchestra.run.smithi098.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:46:08.708 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:46:08.708 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-29T11:46:08.709 INFO:teuthology.orchestra.run.smithi098.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:46:08.709 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:46:08.709 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:46:08.709 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:46:08.709 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:46:08.709 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:46:08.709 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:46:08.709 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:46:08.709 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:46:08.709 INFO:teuthology.orchestra.run.smithi098.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:46:08.709 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:46:08.709 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:46:08.709 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:46:08.709 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:46:08.709 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:46:08.710 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:46:08.710 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:46:08.710 INFO:teuthology.orchestra.run.smithi098.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:46:08.710 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:46:08.710 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:46:08.710 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:46:08.710 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:46:08.710 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:46:08.710 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:46:08.710 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:46:08.710 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:46:08.710 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:46:08.710 INFO:teuthology.orchestra.run.smithi098.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:46:08.710 INFO:teuthology.orchestra.run.smithi098.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:46:08.710 INFO:teuthology.orchestra.run.smithi098.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:46:08.711 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:46:08.711 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T11:46:08.831 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:11 ago on Fri 29 Mar 2024 11:44:57 AM UTC. 2024-03-29T11:46:08.960 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-test 2024-03-29T11:46:08.981 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:46:08.981 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:46:08.981 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-03-29T11:46:08.982 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:46:08.982 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-29T11:46:08.982 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:46:08.982 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:08.982 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-29T11:46:08.982 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:46:08.982 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-03-29T11:46:08.982 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:08.982 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 747 k 2024-03-29T11:46:08.983 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 754 k 2024-03-29T11:46:08.983 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-29T11:46:09.517 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:45 ago on Fri 29 Mar 2024 11:45:24 AM UTC. 2024-03-29T11:46:09.671 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T11:46:09.672 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T11:46:09.672 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-29T11:46:09.672 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T11:46:09.672 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-29T11:46:09.672 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:46:09.672 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-29T11:46:09.672 INFO:teuthology.orchestra.run.smithi098.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:46:09.672 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:46:09.672 INFO:teuthology.orchestra.run.smithi098.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:46:09.673 INFO:teuthology.orchestra.run.smithi098.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:46:09.673 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:46:09.673 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:46:09.673 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-29T11:46:09.673 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T11:46:09.673 INFO:teuthology.orchestra.run.smithi098.stdout:Install 6 Packages 2024-03-29T11:46:09.673 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:46:09.673 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 48 M 2024-03-29T11:46:09.674 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 204 M 2024-03-29T11:46:09.674 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-29T11:46:09.695 INFO:teuthology.orchestra.run.smithi121.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.0 MB/s | 747 kB 00:00 2024-03-29T11:46:09.696 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:46:09.696 INFO:teuthology.orchestra.run.smithi121.stdout:Total 1.0 MB/s | 747 kB 00:00 2024-03-29T11:46:09.696 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-29T11:46:09.701 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-29T11:46:09.702 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-29T11:46:09.708 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-29T11:46:09.708 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-29T11:46:09.785 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-29T11:46:10.081 INFO:teuthology.orchestra.run.smithi098.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-03-29T11:46:10.123 INFO:teuthology.orchestra.run.smithi098.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-03-29T11:46:10.149 INFO:teuthology.orchestra.run.smithi098.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.5 MB/s | 63 kB 00:00 2024-03-29T11:46:10.187 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:46:10.276 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:46:10.290 INFO:teuthology.orchestra.run.smithi098.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-03-29T11:46:10.566 INFO:teuthology.orchestra.run.smithi098.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 258 kB/s | 163 kB 00:00 2024-03-29T11:46:10.675 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:46:10.951 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:46:10.951 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:10.951 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-29T11:46:10.952 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:46:10.952 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:10.952 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:46:11.161 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:46:11.737 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:14 ago on Fri 29 Mar 2024 11:44:57 AM UTC. 2024-03-29T11:46:11.894 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:46:11.895 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:46:11.895 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-03-29T11:46:11.895 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:46:11.895 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-29T11:46:11.895 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:46:11.895 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:11.895 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-29T11:46:11.895 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:46:11.895 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-03-29T11:46:11.895 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:11.896 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 145 k 2024-03-29T11:46:11.896 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 436 k 2024-03-29T11:46:11.896 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-29T11:46:12.478 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 250 kB/s | 145 kB 00:00 2024-03-29T11:46:12.478 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:46:12.479 INFO:teuthology.orchestra.run.smithi121.stdout:Total 249 kB/s | 145 kB 00:00 2024-03-29T11:46:12.479 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-29T11:46:12.486 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-29T11:46:12.486 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-29T11:46:12.539 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-29T11:46:12.539 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-29T11:46:12.691 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-29T11:46:12.787 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:46:12.812 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:46:12.812 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:46:12.812 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:46:12.812 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:13.745 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:46:13.745 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:13.745 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-29T11:46:13.745 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:46:13.745 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:13.745 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:46:13.991 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr 2024-03-29T11:46:14.561 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:17 ago on Fri 29 Mar 2024 11:44:57 AM UTC. 2024-03-29T11:46:14.659 INFO:teuthology.orchestra.run.smithi121.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:46:14.711 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:46:14.712 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-03-29T11:46:14.712 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:46:14.785 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:46:15.341 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:18 ago on Fri 29 Mar 2024 11:44:57 AM UTC. 2024-03-29T11:46:15.441 INFO:teuthology.orchestra.run.smithi098.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 8.6 MB/s | 47 MB 00:05 2024-03-29T11:46:15.444 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:46:15.444 INFO:teuthology.orchestra.run.smithi098.stdout:Total 8.3 MB/s | 48 MB 00:05 2024-03-29T11:46:15.458 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-29T11:46:15.478 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-29T11:46:15.478 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-29T11:46:15.493 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:46:15.494 INFO:teuthology.orchestra.run.smithi121.stdout:====================================================================================== 2024-03-29T11:46:15.494 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-03-29T11:46:15.494 INFO:teuthology.orchestra.run.smithi121.stdout:====================================================================================== 2024-03-29T11:46:15.494 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-29T11:46:15.494 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:46:15.494 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-03-29T11:46:15.495 INFO:teuthology.orchestra.run.smithi121.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:46:15.495 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:46:15.495 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:46:15.495 INFO:teuthology.orchestra.run.smithi121.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:46:15.495 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:46:15.495 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:46:15.495 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:46:15.495 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:46:15.495 INFO:teuthology.orchestra.run.smithi121.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:46:15.495 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:46:15.495 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:46:15.495 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:15.495 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-29T11:46:15.495 INFO:teuthology.orchestra.run.smithi121.stdout:====================================================================================== 2024-03-29T11:46:15.496 INFO:teuthology.orchestra.run.smithi121.stdout:Install 12 Packages 2024-03-29T11:46:15.496 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:15.496 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 6.9 M 2024-03-29T11:46:15.496 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 26 M 2024-03-29T11:46:15.496 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-29T11:46:15.832 INFO:teuthology.orchestra.run.smithi121.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 162 kB/s | 31 kB 00:00 2024-03-29T11:46:15.857 INFO:teuthology.orchestra.run.smithi121.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 70 kB/s | 15 kB 00:00 2024-03-29T11:46:15.859 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-29T11:46:15.859 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-29T11:46:15.991 INFO:teuthology.orchestra.run.smithi121.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.7 MB/s | 238 kB 00:00 2024-03-29T11:46:16.033 INFO:teuthology.orchestra.run.smithi121.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.0 MB/s | 1.0 MB 00:00 2024-03-29T11:46:16.158 INFO:teuthology.orchestra.run.smithi121.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 3.4 MB/s | 1.8 MB 00:00 2024-03-29T11:46:16.242 INFO:teuthology.orchestra.run.smithi121.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.4 MB/s | 863 kB 00:00 2024-03-29T11:46:16.267 INFO:teuthology.orchestra.run.smithi121.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 179 kB/s | 19 kB 00:00 2024-03-29T11:46:16.376 INFO:teuthology.orchestra.run.smithi121.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.6 MB/s | 548 kB 00:00 2024-03-29T11:46:16.409 INFO:teuthology.orchestra.run.smithi121.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.0 MB/s | 144 kB 00:00 2024-03-29T11:46:16.443 INFO:teuthology.orchestra.run.smithi121.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 457 kB/s | 31 kB 00:00 2024-03-29T11:46:16.446 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-29T11:46:16.477 INFO:teuthology.orchestra.run.smithi121.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.7 MB/s | 188 kB 00:00 2024-03-29T11:46:16.516 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:46:16.555 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:46:16.596 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:46:16.641 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:46:16.669 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:46:16.727 INFO:teuthology.orchestra.run.smithi121.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 4.2 MB/s | 2.0 MB 00:00 2024-03-29T11:46:16.730 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:46:16.730 INFO:teuthology.orchestra.run.smithi121.stdout:Total 5.6 MB/s | 6.9 MB 00:01 2024-03-29T11:46:16.748 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-29T11:46:16.765 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-29T11:46:16.765 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-29T11:46:16.939 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-29T11:46:16.939 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-29T11:46:17.456 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-29T11:46:17.527 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:46:17.562 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:46:17.609 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:46:17.640 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:46:17.747 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:46:17.872 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:46:17.966 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:46:18.011 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:46:18.040 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:46:18.071 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:46:18.313 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:46:18.346 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:46:19.136 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:46:19.136 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:46:19.136 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:46:19.136 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:46:19.136 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:46:19.136 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:46:19.136 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:46:19.136 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:46:19.136 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:46:19.137 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:46:19.137 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:46:19.137 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:46:19.479 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:46:19.480 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:19.480 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-29T11:46:19.480 INFO:teuthology.orchestra.run.smithi121.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:46:19.480 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:46:19.480 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:46:19.480 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:46:19.480 INFO:teuthology.orchestra.run.smithi121.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:46:19.480 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:46:19.480 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:46:19.480 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:46:19.480 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:46:19.480 INFO:teuthology.orchestra.run.smithi121.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:46:19.480 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:46:19.481 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:46:19.481 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:19.481 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:46:19.779 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:46:20.336 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:23 ago on Fri 29 Mar 2024 11:44:57 AM UTC. 2024-03-29T11:46:20.488 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:46:20.489 INFO:teuthology.orchestra.run.smithi121.stdout:============================================================================================ 2024-03-29T11:46:20.489 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-03-29T11:46:20.489 INFO:teuthology.orchestra.run.smithi121.stdout:============================================================================================ 2024-03-29T11:46:20.489 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-29T11:46:20.489 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:46:20.490 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-03-29T11:46:20.490 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:46:20.490 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:46:20.490 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:46:20.490 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:46:20.490 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:46:20.490 INFO:teuthology.orchestra.run.smithi121.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:46:20.490 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:46:20.490 INFO:teuthology.orchestra.run.smithi121.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:46:20.490 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:46:20.490 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:46:20.490 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:46:20.490 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:20.490 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-29T11:46:20.491 INFO:teuthology.orchestra.run.smithi121.stdout:============================================================================================ 2024-03-29T11:46:20.491 INFO:teuthology.orchestra.run.smithi121.stdout:Install 12 Packages 2024-03-29T11:46:20.491 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:20.491 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 37 M 2024-03-29T11:46:20.491 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 209 M 2024-03-29T11:46:20.491 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-29T11:46:20.884 INFO:teuthology.orchestra.run.smithi121.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 211 kB/s | 32 kB 00:00 2024-03-29T11:46:20.918 INFO:teuthology.orchestra.run.smithi121.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 534 kB/s | 17 kB 00:00 2024-03-29T11:46:20.960 INFO:teuthology.orchestra.run.smithi121.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 850 kB/s | 34 kB 00:00 2024-03-29T11:46:21.010 INFO:teuthology.orchestra.run.smithi121.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-29T11:46:21.128 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:46:21.219 INFO:teuthology.orchestra.run.smithi121.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2024-03-29T11:46:21.219 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:46:21.286 INFO:teuthology.orchestra.run.smithi121.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 4.8 MB 00:00 2024-03-29T11:46:21.311 INFO:teuthology.orchestra.run.smithi121.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-29T11:46:21.470 INFO:teuthology.orchestra.run.smithi121.stdout:(8/12): ceph-mgr-diskprediction-local-19.0.0-14 10 MB/s | 7.4 MB 00:00 2024-03-29T11:46:21.670 INFO:teuthology.orchestra.run.smithi121.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-03-29T11:46:21.728 INFO:teuthology.orchestra.run.smithi121.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 762 kB/s | 195 kB 00:00 2024-03-29T11:46:21.754 INFO:teuthology.orchestra.run.smithi121.stdout:(11/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.8 MB/s | 803 kB 00:00 2024-03-29T11:46:21.937 INFO:teuthology.orchestra.run.smithi121.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 920 kB/s | 245 kB 00:00 2024-03-29T11:46:21.940 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:46:21.940 INFO:teuthology.orchestra.run.smithi121.stdout:Total 26 MB/s | 37 MB 00:01 2024-03-29T11:46:21.980 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-29T11:46:22.003 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-29T11:46:22.004 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-29T11:46:22.360 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-29T11:46:22.360 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-29T11:46:23.084 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-29T11:46:23.193 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:46:23.233 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:46:23.325 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:46:23.361 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:46:23.532 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:46:23.560 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:46:23.572 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:46:23.572 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:46:23.572 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:46:23.572 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:46:23.573 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:46:23.573 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:46:23.647 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:46:23.984 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:46:23.984 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:46:23.984 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-29T11:46:23.984 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:46:23.984 INFO:teuthology.orchestra.run.smithi098.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:46:23.984 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:46:23.984 INFO:teuthology.orchestra.run.smithi098.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:46:23.984 INFO:teuthology.orchestra.run.smithi098.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:46:23.984 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:46:23.984 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:46:23.984 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T11:46:24.286 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph 2024-03-29T11:46:24.317 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:46:24.439 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:46:24.850 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:00 ago on Fri 29 Mar 2024 11:45:24 AM UTC. 2024-03-29T11:46:25.015 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T11:46:25.017 INFO:teuthology.orchestra.run.smithi098.stdout:========================================================================================= 2024-03-29T11:46:25.017 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-29T11:46:25.017 INFO:teuthology.orchestra.run.smithi098.stdout:========================================================================================= 2024-03-29T11:46:25.017 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-29T11:46:25.017 INFO:teuthology.orchestra.run.smithi098.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:46:25.017 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-29T11:46:25.018 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:46:25.018 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:46:25.018 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:46:25.018 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:46:25.018 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:46:25.018 INFO:teuthology.orchestra.run.smithi098.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:46:25.018 INFO:teuthology.orchestra.run.smithi098.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:46:25.018 INFO:teuthology.orchestra.run.smithi098.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:46:25.018 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:46:25.018 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:46:25.018 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:46:25.018 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:46:25.019 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:46:25.019 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:46:25.019 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:46:25.019 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:46:25.019 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:46:25.019 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:46:25.019 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:46:25.019 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:46:25.019 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:46:25.019 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:46:25.019 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:46:25.019 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:46:25.019 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:46:25.019 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:46:25.020 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:46:25.020 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:46:25.020 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:46:25.020 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:46:25.020 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:46:25.020 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:46:25.020 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:46:25.020 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:46:25.020 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:46:25.020 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:46:25.020 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-29T11:46:25.020 INFO:teuthology.orchestra.run.smithi098.stdout:========================================================================================= 2024-03-29T11:46:25.020 INFO:teuthology.orchestra.run.smithi098.stdout:Install 36 Packages 2024-03-29T11:46:25.020 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:46:25.021 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 30 M 2024-03-29T11:46:25.021 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 107 M 2024-03-29T11:46:25.021 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-29T11:46:25.604 INFO:teuthology.orchestra.run.smithi098.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 36 kB/s | 6.6 kB 00:00 2024-03-29T11:46:25.679 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:46:25.787 INFO:teuthology.orchestra.run.smithi098.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 5.8 MB/s | 2.1 MB 00:00 2024-03-29T11:46:26.188 INFO:teuthology.orchestra.run.smithi098.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 1.9 MB/s | 1.5 MB 00:00 2024-03-29T11:46:26.305 INFO:teuthology.orchestra.run.smithi098.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 2.1 MB/s | 247 kB 00:00 2024-03-29T11:46:26.413 INFO:teuthology.orchestra.run.smithi098.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2024-03-29T11:46:26.455 INFO:teuthology.orchestra.run.smithi098.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T11:46:26.481 INFO:teuthology.orchestra.run.smithi098.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-29T11:46:26.514 INFO:teuthology.orchestra.run.smithi098.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T11:46:26.548 INFO:teuthology.orchestra.run.smithi098.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-29T11:46:26.703 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:46:26.706 INFO:teuthology.orchestra.run.smithi098.stdout:(10/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 18 MB/s | 17 MB 00:00 2024-03-29T11:46:26.732 INFO:teuthology.orchestra.run.smithi098.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 241 kB 00:00 2024-03-29T11:46:26.735 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:46:26.799 INFO:teuthology.orchestra.run.smithi098.stdout:(12/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 3.9 MB/s | 4.7 MB 00:01 2024-03-29T11:46:26.832 INFO:teuthology.orchestra.run.smithi098.stdout:(13/36): python3-cryptography-36.0.1-4.el9.x86_ 9.2 MB/s | 1.1 MB 00:00 2024-03-29T11:46:26.858 INFO:teuthology.orchestra.run.smithi098.stdout:(14/36): python3-ply-3.11-14.el9.noarch.rpm 825 kB/s | 103 kB 00:00 2024-03-29T11:46:26.883 INFO:teuthology.orchestra.run.smithi098.stdout:(15/36): python3-pycparser-2.20-6.el9.noarch.rp 1.5 MB/s | 124 kB 00:00 2024-03-29T11:46:26.908 INFO:teuthology.orchestra.run.smithi098.stdout:(16/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-29T11:46:27.050 INFO:teuthology.orchestra.run.smithi098.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 134 kB/s | 22 kB 00:00 2024-03-29T11:46:27.301 INFO:teuthology.orchestra.run.smithi098.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 385 kB/s | 151 kB 00:00 2024-03-29T11:46:27.329 INFO:teuthology.orchestra.run.smithi098.stdout:(19/36): python3-requests-2.25.1-8.el9.noarch.r 267 kB/s | 125 kB 00:00 2024-03-29T11:46:27.351 INFO:teuthology.orchestra.run.smithi098.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 144 kB/s | 43 kB 00:00 2024-03-29T11:46:27.443 INFO:teuthology.orchestra.run.smithi098.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 117 kB/s | 11 kB 00:00 2024-03-29T11:46:27.468 INFO:teuthology.orchestra.run.smithi098.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-03-29T11:46:27.535 INFO:teuthology.orchestra.run.smithi098.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 192 kB/s | 18 kB 00:00 2024-03-29T11:46:27.569 INFO:teuthology.orchestra.run.smithi098.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 231 kB/s | 23 kB 00:00 2024-03-29T11:46:27.619 INFO:teuthology.orchestra.run.smithi098.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 233 kB/s | 19 kB 00:00 2024-03-29T11:46:27.652 INFO:teuthology.orchestra.run.smithi098.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 235 kB/s | 20 kB 00:00 2024-03-29T11:46:27.703 INFO:teuthology.orchestra.run.smithi098.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 553 kB/s | 46 kB 00:00 2024-03-29T11:46:27.753 INFO:teuthology.orchestra.run.smithi098.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 785 kB/s | 79 kB 00:00 2024-03-29T11:46:27.845 INFO:teuthology.orchestra.run.smithi098.stdout:(29/36): python3-cherrypy-18.6.1-2.el9.noarch.r 690 kB/s | 358 kB 00:00 2024-03-29T11:46:27.862 INFO:teuthology.orchestra.run.smithi098.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 151 kB/s | 16 kB 00:00 2024-03-29T11:46:27.939 INFO:teuthology.orchestra.run.smithi098.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 981 kB/s | 90 kB 00:00 2024-03-29T11:46:27.963 INFO:teuthology.orchestra.run.smithi098.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 356 kB/s | 36 kB 00:00 2024-03-29T11:46:28.030 INFO:teuthology.orchestra.run.smithi098.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-03-29T11:46:28.122 INFO:teuthology.orchestra.run.smithi098.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 218 kB/s | 20 kB 00:00 2024-03-29T11:46:28.146 INFO:teuthology.orchestra.run.smithi098.stdout:(35/36): python3-pecan-1.4.2-3.el9.noarch.rpm 614 kB/s | 272 kB 00:00 2024-03-29T11:46:28.205 INFO:teuthology.orchestra.run.smithi098.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.7 MB/s | 427 kB 00:00 2024-03-29T11:46:28.207 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:46:28.207 INFO:teuthology.orchestra.run.smithi098.stdout:Total 9.3 MB/s | 30 MB 00:03 2024-03-29T11:46:28.231 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-29T11:46:28.274 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-29T11:46:28.274 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-29T11:46:28.682 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-29T11:46:28.683 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-29T11:46:29.276 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:46:29.276 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:46:29.276 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:46:29.276 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:46:29.276 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:46:29.276 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:46:29.276 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:46:29.276 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:46:29.276 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:46:29.277 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:46:29.277 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:46:29.277 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:46:29.665 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:46:29.666 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:29.666 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-29T11:46:29.666 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:46:29.666 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:46:29.666 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:46:29.666 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:46:29.666 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:46:29.666 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:46:29.666 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:46:29.666 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:46:29.666 INFO:teuthology.orchestra.run.smithi121.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:46:29.666 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:46:29.666 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:46:29.667 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:46:29.667 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:29.667 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:46:30.006 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:46:30.012 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-29T11:46:30.083 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:46:30.125 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:46:30.161 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:46:30.230 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:46:30.268 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:46:30.301 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:46:30.392 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:46:30.470 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:46:30.512 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:46:30.577 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:33 ago on Fri 29 Mar 2024 11:44:57 AM UTC. 2024-03-29T11:46:30.583 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:46:30.637 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:46:30.676 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:46:30.725 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:46:30.728 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:46:30.729 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================== 2024-03-29T11:46:30.729 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-03-29T11:46:30.729 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================== 2024-03-29T11:46:30.730 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-29T11:46:30.730 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:46:30.730 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-03-29T11:46:30.730 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:46:30.730 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:46:30.730 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:46:30.730 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:46:30.730 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:46:30.730 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:46:30.730 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:46:30.730 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:46:30.730 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:46:30.730 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:46:30.731 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:46:30.731 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:46:30.731 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:30.731 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-29T11:46:30.731 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================== 2024-03-29T11:46:30.731 INFO:teuthology.orchestra.run.smithi121.stdout:Install 13 Packages 2024-03-29T11:46:30.731 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:30.731 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 2.0 M 2024-03-29T11:46:30.732 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 26 M 2024-03-29T11:46:30.732 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-29T11:46:30.779 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:46:30.972 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:46:31.022 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:46:31.052 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:46:31.087 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:46:31.117 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:46:31.144 INFO:teuthology.orchestra.run.smithi121.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 125 kB/s | 24 kB 00:00 2024-03-29T11:46:31.169 INFO:teuthology.orchestra.run.smithi121.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 82 kB/s | 18 kB 00:00 2024-03-29T11:46:31.194 INFO:teuthology.orchestra.run.smithi121.stdout:(3/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 213 kB/s | 52 kB 00:00 2024-03-29T11:46:31.201 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:46:31.244 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:46:31.253 INFO:teuthology.orchestra.run.smithi121.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.6 MB/s | 174 kB 00:00 2024-03-29T11:46:31.278 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:46:31.278 INFO:teuthology.orchestra.run.smithi121.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 132 kB 00:00 2024-03-29T11:46:31.303 INFO:teuthology.orchestra.run.smithi121.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.9 MB/s | 211 kB 00:00 2024-03-29T11:46:31.318 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:46:31.329 INFO:teuthology.orchestra.run.smithi121.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 571 kB/s | 43 kB 00:00 2024-03-29T11:46:31.353 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:46:31.380 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:46:31.462 INFO:teuthology.orchestra.run.smithi121.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 203 kB/s | 32 kB 00:00 2024-03-29T11:46:31.467 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:46:31.502 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:46:31.574 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:46:31.588 INFO:teuthology.orchestra.run.smithi121.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 784 kB/s | 203 kB 00:00 2024-03-29T11:46:31.688 INFO:teuthology.orchestra.run.smithi121.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 591 kB/s | 59 kB 00:00 2024-03-29T11:46:31.710 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:46:31.747 INFO:teuthology.orchestra.run.smithi121.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 3.6 MB/s | 1.0 MB 00:00 2024-03-29T11:46:31.772 INFO:teuthology.orchestra.run.smithi121.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.1 MB/s | 90 kB 00:00 2024-03-29T11:46:31.846 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:46:32.020 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:46:32.049 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:46:32.073 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:46:32.074 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:46:32.074 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:46:32.074 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:46:32.074 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:46:32.074 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:46:32.480 INFO:teuthology.orchestra.run.smithi121.stdout:(13/13): python3-certifi-2021.10.8-2.el9.noarch 13 kB/s | 15 kB 00:01 2024-03-29T11:46:32.482 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:46:32.482 INFO:teuthology.orchestra.run.smithi121.stdout:Total 1.2 MB/s | 2.0 MB 00:01 2024-03-29T11:46:32.483 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-29T11:46:32.498 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-29T11:46:32.498 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-29T11:46:32.592 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-29T11:46:32.594 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-29T11:46:33.104 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-29T11:46:33.277 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:46:33.334 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:46:33.387 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:46:33.424 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:46:33.465 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:46:33.525 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:46:33.551 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:46:33.551 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:46:33.551 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:46:33.551 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:46:33.551 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:46:33.552 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:46:33.577 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:46:33.686 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:46:33.742 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:46:33.993 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:46:34.016 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:46:34.016 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:46:34.016 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:46:34.016 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:46:34.017 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:46:34.017 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:46:34.235 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:46:34.258 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:46:34.258 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:46:34.258 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:46:34.258 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:46:34.258 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:46:34.258 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:46:34.324 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:46:34.458 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:46:34.514 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:46:34.556 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:46:34.604 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:46:34.626 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:46:35.441 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:46:35.441 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:46:35.441 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:46:35.441 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:46:35.441 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:46:35.441 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:46:35.441 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:46:35.441 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:46:35.441 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:46:35.442 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:46:35.442 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:46:35.442 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:46:35.442 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:46:35.804 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:46:35.804 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:35.804 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-29T11:46:35.804 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:46:35.804 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:46:35.804 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:46:35.805 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:46:35.805 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:46:35.805 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:46:35.805 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:46:35.805 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:46:35.805 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:46:35.805 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:46:35.805 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:46:35.805 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:46:35.805 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:46:35.805 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:35.805 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:46:36.064 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:46:36.064 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:46:36.064 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:46:36.065 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:46:36.065 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:46:36.065 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:46:36.065 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:46:36.065 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:46:36.065 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:46:36.065 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:46:36.065 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:46:36.065 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:46:36.065 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:46:36.065 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:46:36.065 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:46:36.065 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:46:36.065 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:46:36.066 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:46:36.066 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:46:36.066 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:46:36.066 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:46:36.066 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:46:36.066 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:46:36.066 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:46:36.066 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:46:36.066 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:46:36.066 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:46:36.066 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:46:36.066 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:46:36.066 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:46:36.067 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:46:36.067 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:46:36.067 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:46:36.067 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:46:36.067 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:46:36.067 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:46:36.080 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:46:36.644 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 11:44:57 AM UTC. 2024-03-29T11:46:36.795 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:46:36.795 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================== 2024-03-29T11:46:36.796 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-03-29T11:46:36.796 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================== 2024-03-29T11:46:36.796 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-29T11:46:36.796 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:46:36.796 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-03-29T11:46:36.796 INFO:teuthology.orchestra.run.smithi121.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:46:36.796 INFO:teuthology.orchestra.run.smithi121.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:46:36.796 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:46:36.796 INFO:teuthology.orchestra.run.smithi121.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:46:36.796 INFO:teuthology.orchestra.run.smithi121.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:46:36.796 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:36.796 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-29T11:46:36.797 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================== 2024-03-29T11:46:36.797 INFO:teuthology.orchestra.run.smithi121.stdout:Install 6 Packages 2024-03-29T11:46:36.797 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:36.797 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 6.8 M 2024-03-29T11:46:36.797 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 33 M 2024-03-29T11:46:36.797 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-29T11:46:37.054 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:46:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:46:37.055 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-29T11:46:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:46:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:46:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:46:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:46:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:46:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:46:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:46:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:46:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:46:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:46:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:46:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:46:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:46:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:46:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:46:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:46:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:46:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:46:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:46:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:46:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:46:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:46:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:46:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:46:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:46:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:46:37.057 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:46:37.057 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:46:37.057 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:46:37.057 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:46:37.057 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:46:37.057 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:46:37.057 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:46:37.057 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:46:37.057 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:46:37.057 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:46:37.057 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:46:37.057 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T11:46:37.249 INFO:teuthology.orchestra.run.smithi121.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.5 MB/s | 248 kB 00:00 2024-03-29T11:46:37.317 INFO:teuthology.orchestra.run.smithi121.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 599 kB/s | 138 kB 00:00 2024-03-29T11:46:37.368 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-base 2024-03-29T11:46:37.400 INFO:teuthology.orchestra.run.smithi121.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-03-29T11:46:37.675 INFO:teuthology.orchestra.run.smithi121.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 162 kB/s | 58 kB 00:00 2024-03-29T11:46:37.742 INFO:teuthology.orchestra.run.smithi121.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 149 kB/s | 51 kB 00:00 2024-03-29T11:46:37.958 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:13 ago on Fri 29 Mar 2024 11:45:24 AM UTC. 2024-03-29T11:46:37.959 INFO:teuthology.orchestra.run.smithi121.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 775 kB/s | 548 kB 00:00 2024-03-29T11:46:37.961 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:46:37.961 INFO:teuthology.orchestra.run.smithi121.stdout:Total 5.8 MB/s | 6.8 MB 00:01 2024-03-29T11:46:37.974 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-29T11:46:37.984 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-29T11:46:37.984 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-29T11:46:38.062 INFO:teuthology.orchestra.run.smithi098.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:46:38.072 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-29T11:46:38.072 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-29T11:46:38.117 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T11:46:38.118 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-03-29T11:46:38.118 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T11:46:38.194 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install cephadm 2024-03-29T11:46:38.360 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-29T11:46:38.522 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:46:38.585 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:46:38.777 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:14 ago on Fri 29 Mar 2024 11:45:24 AM UTC. 2024-03-29T11:46:38.929 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T11:46:38.930 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T11:46:38.930 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-29T11:46:38.930 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T11:46:38.930 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-29T11:46:38.930 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:46:38.930 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:46:38.930 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-29T11:46:38.930 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T11:46:38.930 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-29T11:46:38.930 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:46:38.931 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 747 k 2024-03-29T11:46:38.931 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 754 k 2024-03-29T11:46:38.931 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-29T11:46:39.100 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:46:39.179 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:46:39.240 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:46:39.264 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:46:39.465 INFO:teuthology.orchestra.run.smithi098.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.4 MB/s | 747 kB 00:00 2024-03-29T11:46:39.465 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:46:39.466 INFO:teuthology.orchestra.run.smithi098.stdout:Total 1.4 MB/s | 747 kB 00:00 2024-03-29T11:46:39.466 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-29T11:46:39.471 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-29T11:46:39.471 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-29T11:46:39.477 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-29T11:46:39.477 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-29T11:46:39.553 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-29T11:46:40.024 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:46:40.083 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:46:40.083 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:46:40.083 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:46:40.084 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:46:40.084 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:46:40.084 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:46:40.111 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:46:40.367 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:46:40.368 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:40.368 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-29T11:46:40.368 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:46:40.368 INFO:teuthology.orchestra.run.smithi121.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:46:40.368 INFO:teuthology.orchestra.run.smithi121.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:46:40.368 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:46:40.368 INFO:teuthology.orchestra.run.smithi121.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:46:40.368 INFO:teuthology.orchestra.run.smithi121.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:46:40.368 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:40.368 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:46:40.527 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:46:40.641 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-fuse 2024-03-29T11:46:40.816 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:46:40.816 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:46:40.816 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-29T11:46:40.816 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:46:40.816 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:46:40.816 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T11:46:41.001 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:46:41.202 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 11:44:57 AM UTC. 2024-03-29T11:46:41.352 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:46:41.353 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:46:41.353 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-03-29T11:46:41.353 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:46:41.353 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-29T11:46:41.353 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:46:41.353 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-03-29T11:46:41.353 INFO:teuthology.orchestra.run.smithi121.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:46:41.353 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:41.353 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-29T11:46:41.353 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:46:41.353 INFO:teuthology.orchestra.run.smithi121.stdout:Install 2 Packages 2024-03-29T11:46:41.353 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:41.353 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 964 k 2024-03-29T11:46:41.354 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 2.8 M 2024-03-29T11:46:41.354 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-29T11:46:41.544 INFO:teuthology.orchestra.run.smithi121.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 412 kB/s | 79 kB 00:00 2024-03-29T11:46:41.569 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:17 ago on Fri 29 Mar 2024 11:45:24 AM UTC. 2024-03-29T11:46:41.619 INFO:teuthology.orchestra.run.smithi121.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 3.3 MB/s | 885 kB 00:00 2024-03-29T11:46:41.621 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:46:41.621 INFO:teuthology.orchestra.run.smithi121.stdout:Total 3.5 MB/s | 964 kB 00:00 2024-03-29T11:46:41.621 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-29T11:46:41.641 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-29T11:46:41.641 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-29T11:46:41.714 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-29T11:46:41.714 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-29T11:46:41.723 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T11:46:41.723 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T11:46:41.723 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-29T11:46:41.724 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T11:46:41.724 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-29T11:46:41.724 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:46:41.724 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:46:41.724 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-29T11:46:41.724 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T11:46:41.724 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-29T11:46:41.724 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:46:41.725 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 145 k 2024-03-29T11:46:41.725 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 436 k 2024-03-29T11:46:41.725 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-29T11:46:41.904 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-29T11:46:41.947 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 655 kB/s | 145 kB 00:00 2024-03-29T11:46:41.947 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:46:41.948 INFO:teuthology.orchestra.run.smithi098.stdout:Total 651 kB/s | 145 kB 00:00 2024-03-29T11:46:41.948 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-29T11:46:41.955 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-29T11:46:41.955 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-29T11:46:42.007 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-29T11:46:42.007 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-29T11:46:42.049 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:46:42.121 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:46:42.141 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-29T11:46:42.190 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:46:42.214 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:46:42.214 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:46:42.214 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:46:42.214 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:46:42.859 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:46:42.860 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:46:43.138 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:46:43.138 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:46:43.138 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-29T11:46:43.138 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:46:43.139 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:46:43.139 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T11:46:43.176 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:46:43.176 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:43.176 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-29T11:46:43.176 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:46:43.176 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:43.176 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:46:43.374 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr 2024-03-29T11:46:43.458 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-volume 2024-03-29T11:46:43.942 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:19 ago on Fri 29 Mar 2024 11:45:24 AM UTC. 2024-03-29T11:46:44.039 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 11:44:57 AM UTC. 2024-03-29T11:46:44.042 INFO:teuthology.orchestra.run.smithi098.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:46:44.094 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T11:46:44.095 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-03-29T11:46:44.095 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T11:46:44.158 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:46:44.191 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:46:44.192 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:46:44.192 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-03-29T11:46:44.193 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:46:44.193 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-29T11:46:44.193 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:46:44.193 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-03-29T11:46:44.193 INFO:teuthology.orchestra.run.smithi121.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:46:44.193 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:44.193 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-29T11:46:44.193 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:46:44.194 INFO:teuthology.orchestra.run.smithi121.stdout:Install 2 Packages 2024-03-29T11:46:44.194 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:44.194 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 332 k 2024-03-29T11:46:44.194 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 1.5 M 2024-03-29T11:46:44.194 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-29T11:46:44.319 INFO:teuthology.orchestra.run.smithi121.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 548 kB/s | 69 kB 00:00 2024-03-29T11:46:44.720 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:20 ago on Fri 29 Mar 2024 11:45:24 AM UTC. 2024-03-29T11:46:44.835 INFO:teuthology.orchestra.run.smithi121.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 410 kB/s | 264 kB 00:00 2024-03-29T11:46:44.836 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:46:44.837 INFO:teuthology.orchestra.run.smithi121.stdout:Total 515 kB/s | 332 kB 00:00 2024-03-29T11:46:44.838 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-29T11:46:44.846 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-29T11:46:44.847 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-29T11:46:44.870 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T11:46:44.871 INFO:teuthology.orchestra.run.smithi098.stdout:====================================================================================== 2024-03-29T11:46:44.872 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-29T11:46:44.872 INFO:teuthology.orchestra.run.smithi098.stdout:====================================================================================== 2024-03-29T11:46:44.872 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-29T11:46:44.872 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:46:44.872 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-29T11:46:44.872 INFO:teuthology.orchestra.run.smithi098.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:46:44.872 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:46:44.872 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:46:44.872 INFO:teuthology.orchestra.run.smithi098.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:46:44.872 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:46:44.872 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:46:44.872 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:46:44.872 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:46:44.873 INFO:teuthology.orchestra.run.smithi098.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:46:44.873 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:46:44.873 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:46:44.873 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:46:44.873 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-29T11:46:44.873 INFO:teuthology.orchestra.run.smithi098.stdout:====================================================================================== 2024-03-29T11:46:44.873 INFO:teuthology.orchestra.run.smithi098.stdout:Install 12 Packages 2024-03-29T11:46:44.873 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:46:44.874 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 6.9 M 2024-03-29T11:46:44.874 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 26 M 2024-03-29T11:46:44.874 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-29T11:46:44.887 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-29T11:46:44.887 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-29T11:46:45.038 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-29T11:46:45.218 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:46:45.255 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:46:45.272 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:46:45.272 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:46:45.272 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:46:45.272 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:45.402 INFO:teuthology.orchestra.run.smithi098.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 87 kB/s | 31 kB 00:00 2024-03-29T11:46:45.527 INFO:teuthology.orchestra.run.smithi098.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 31 kB/s | 15 kB 00:00 2024-03-29T11:46:45.594 INFO:teuthology.orchestra.run.smithi098.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.2 MB/s | 1.0 MB 00:00 2024-03-29T11:46:45.619 INFO:teuthology.orchestra.run.smithi098.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.5 MB/s | 238 kB 00:00 2024-03-29T11:46:45.903 INFO:teuthology.orchestra.run.smithi098.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.9 MB/s | 548 kB 00:00 2024-03-29T11:46:45.953 INFO:teuthology.orchestra.run.smithi098.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 388 kB/s | 19 kB 00:00 2024-03-29T11:46:45.996 INFO:teuthology.orchestra.run.smithi098.stdout:(7/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 1.9 MB/s | 1.8 MB 00:00 2024-03-29T11:46:46.016 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:46:46.037 INFO:teuthology.orchestra.run.smithi098.stdout:(8/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.9 MB/s | 863 kB 00:00 2024-03-29T11:46:46.114 INFO:teuthology.orchestra.run.smithi098.stdout:(9/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 13 MB/s | 2.0 MB 00:00 2024-03-29T11:46:46.163 INFO:teuthology.orchestra.run.smithi098.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 245 kB/s | 31 kB 00:00 2024-03-29T11:46:46.190 INFO:teuthology.orchestra.run.smithi098.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.4 MB/s | 188 kB 00:00 2024-03-29T11:46:46.214 INFO:teuthology.orchestra.run.smithi098.stdout:(12/12): python3-grpcio-tools-1.46.7-10.el9.x86 664 kB/s | 144 kB 00:00 2024-03-29T11:46:46.216 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:46:46.216 INFO:teuthology.orchestra.run.smithi098.stdout:Total 5.1 MB/s | 6.9 MB 00:01 2024-03-29T11:46:46.234 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-29T11:46:46.251 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-29T11:46:46.251 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-29T11:46:46.360 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:46:46.360 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:46.360 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-29T11:46:46.360 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:46:46.360 INFO:teuthology.orchestra.run.smithi121.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:46:46.360 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:46.360 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:46:46.420 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-29T11:46:46.421 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-29T11:46:46.721 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librados-devel 2024-03-29T11:46:46.933 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-29T11:46:47.024 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:46:47.058 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:46:47.104 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:46:47.160 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:46:47.259 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:46:47.308 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:50 ago on Fri 29 Mar 2024 11:44:57 AM UTC. 2024-03-29T11:46:47.385 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:46:47.459 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:46:47.459 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:46:47.459 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-03-29T11:46:47.459 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:46:47.459 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-29T11:46:47.459 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:46:47.459 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:47.459 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-29T11:46:47.459 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:46:47.459 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-03-29T11:46:47.460 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:47.460 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 127 k 2024-03-29T11:46:47.460 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 456 k 2024-03-29T11:46:47.460 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-29T11:46:47.472 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:46:47.540 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:46:47.568 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:46:47.600 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:46:47.841 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:46:47.875 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:46:47.933 INFO:teuthology.orchestra.run.smithi121.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 268 kB/s | 127 kB 00:00 2024-03-29T11:46:47.933 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:46:47.934 INFO:teuthology.orchestra.run.smithi121.stdout:Total 267 kB/s | 127 kB 00:00 2024-03-29T11:46:47.934 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-29T11:46:47.939 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-29T11:46:47.940 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-29T11:46:47.995 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-29T11:46:47.995 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-29T11:46:48.139 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-29T11:46:48.248 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:46:48.639 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:46:48.639 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:46:48.639 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:46:48.639 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:46:48.639 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:46:48.639 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:46:48.639 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:46:48.639 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:46:48.640 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:46:48.640 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:46:48.640 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:46:48.640 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:46:48.685 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:46:48.992 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:46:48.992 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:46:48.992 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-29T11:46:48.993 INFO:teuthology.orchestra.run.smithi098.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:46:48.993 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:46:48.993 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:46:48.993 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:46:48.994 INFO:teuthology.orchestra.run.smithi098.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:46:48.994 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:46:48.994 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:46:48.994 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:46:48.994 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:46:48.995 INFO:teuthology.orchestra.run.smithi098.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:46:48.995 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:46:48.995 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:46:48.995 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:46:48.995 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T11:46:49.017 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:46:49.017 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:49.018 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-29T11:46:49.018 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:46:49.018 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:49.018 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:46:49.261 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:46:49.269 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install libcephfs2 2024-03-29T11:46:49.820 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:25 ago on Fri 29 Mar 2024 11:45:24 AM UTC. 2024-03-29T11:46:49.845 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:52 ago on Fri 29 Mar 2024 11:44:57 AM UTC. 2024-03-29T11:46:49.948 INFO:teuthology.orchestra.run.smithi121.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:46:49.971 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T11:46:49.972 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================================ 2024-03-29T11:46:49.972 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-29T11:46:49.973 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================================ 2024-03-29T11:46:49.973 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-29T11:46:49.973 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:46:49.973 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-29T11:46:49.973 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:46:49.973 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:46:49.973 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:46:49.973 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:46:49.973 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:46:49.973 INFO:teuthology.orchestra.run.smithi098.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:46:49.973 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:46:49.974 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:46:49.974 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:46:49.974 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:46:49.974 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:46:49.974 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:46:49.974 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-29T11:46:49.974 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================================ 2024-03-29T11:46:49.974 INFO:teuthology.orchestra.run.smithi098.stdout:Install 12 Packages 2024-03-29T11:46:49.974 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:46:49.975 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 37 M 2024-03-29T11:46:49.975 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 209 M 2024-03-29T11:46:49.975 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-29T11:46:50.004 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:46:50.005 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-03-29T11:46:50.005 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:46:50.085 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install libcephfs-devel 2024-03-29T11:46:50.337 INFO:teuthology.orchestra.run.smithi098.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 243 kB/s | 32 kB 00:00 2024-03-29T11:46:50.380 INFO:teuthology.orchestra.run.smithi098.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 411 kB/s | 17 kB 00:00 2024-03-29T11:46:50.421 INFO:teuthology.orchestra.run.smithi098.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 843 kB/s | 34 kB 00:00 2024-03-29T11:46:50.605 INFO:teuthology.orchestra.run.smithi098.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 26 MB/s | 4.7 MB 00:00 2024-03-29T11:46:50.652 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:53 ago on Fri 29 Mar 2024 11:44:57 AM UTC. 2024-03-29T11:46:50.739 INFO:teuthology.orchestra.run.smithi098.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-29T11:46:50.781 INFO:teuthology.orchestra.run.smithi098.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.2 MB/s | 217 kB 00:00 2024-03-29T11:46:50.805 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:46:50.806 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:46:50.806 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-03-29T11:46:50.806 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:46:50.806 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-29T11:46:50.807 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:46:50.807 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:50.807 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-29T11:46:50.807 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:46:50.807 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-03-29T11:46:50.807 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:50.807 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 31 k 2024-03-29T11:46:50.807 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 140 k 2024-03-29T11:46:50.807 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-29T11:46:50.857 INFO:teuthology.orchestra.run.smithi098.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.6 MB/s | 3.0 MB 00:00 2024-03-29T11:46:51.062 INFO:teuthology.orchestra.run.smithi121.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 122 kB/s | 31 kB 00:00 2024-03-29T11:46:51.062 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:46:51.062 INFO:teuthology.orchestra.run.smithi121.stdout:Total 121 kB/s | 31 kB 00:00 2024-03-29T11:46:51.063 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-29T11:46:51.066 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-29T11:46:51.066 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-29T11:46:51.092 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-29T11:46:51.093 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-29T11:46:51.222 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-29T11:46:51.240 INFO:teuthology.orchestra.run.smithi098.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 34 MB/s | 16 MB 00:00 2024-03-29T11:46:51.326 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:46:51.465 INFO:teuthology.orchestra.run.smithi098.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 869 kB/s | 195 kB 00:00 2024-03-29T11:46:51.573 INFO:teuthology.orchestra.run.smithi098.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 803 kB 00:00 2024-03-29T11:46:51.674 INFO:teuthology.orchestra.run.smithi098.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-03-29T11:46:51.754 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:46:52.013 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:46:52.013 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:52.014 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-29T11:46:52.014 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:46:52.014 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:52.014 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:46:52.211 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librados2 2024-03-29T11:46:52.773 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:55 ago on Fri 29 Mar 2024 11:44:57 AM UTC. 2024-03-29T11:46:52.824 INFO:teuthology.orchestra.run.smithi098.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 2.8 MB/s | 7.4 MB 00:02 2024-03-29T11:46:52.826 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:46:52.826 INFO:teuthology.orchestra.run.smithi098.stdout:Total 13 MB/s | 37 MB 00:02 2024-03-29T11:46:52.870 INFO:teuthology.orchestra.run.smithi121.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:46:52.870 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-29T11:46:52.896 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-29T11:46:52.896 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-29T11:46:52.922 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:46:52.923 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-03-29T11:46:52.923 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:46:52.976 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librbd1 2024-03-29T11:46:53.254 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-29T11:46:53.254 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-29T11:46:53.545 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:56 ago on Fri 29 Mar 2024 11:44:57 AM UTC. 2024-03-29T11:46:53.643 INFO:teuthology.orchestra.run.smithi121.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:46:53.694 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:46:53.695 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-03-29T11:46:53.695 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:46:53.752 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rados 2024-03-29T11:46:53.959 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-29T11:46:54.055 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:46:54.087 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:46:54.179 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:46:54.223 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:46:54.310 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:57 ago on Fri 29 Mar 2024 11:44:57 AM UTC. 2024-03-29T11:46:54.400 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:46:54.409 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:46:54.430 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:46:54.463 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:46:54.463 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-03-29T11:46:54.464 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:46:54.512 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:46:54.518 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rgw 2024-03-29T11:46:55.081 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:58 ago on Fri 29 Mar 2024 11:44:57 AM UTC. 2024-03-29T11:46:55.174 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:46:55.179 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:46:55.231 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:46:55.232 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-03-29T11:46:55.232 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:46:55.285 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-cephfs 2024-03-29T11:46:55.285 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:46:55.852 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:58 ago on Fri 29 Mar 2024 11:44:57 AM UTC. 2024-03-29T11:46:55.949 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:46:56.000 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:46:56.002 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-03-29T11:46:56.002 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:46:56.045 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rbd 2024-03-29T11:46:56.513 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:46:56.608 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:59 ago on Fri 29 Mar 2024 11:44:57 AM UTC. 2024-03-29T11:46:56.707 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:46:56.757 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:46:56.758 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-03-29T11:46:56.758 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:46:56.813 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-fuse 2024-03-29T11:46:57.377 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:00 ago on Fri 29 Mar 2024 11:44:57 AM UTC. 2024-03-29T11:46:57.524 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:46:57.525 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:46:57.525 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-03-29T11:46:57.525 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:46:57.525 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-29T11:46:57.525 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:46:57.525 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:57.525 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-29T11:46:57.525 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:46:57.525 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-03-29T11:46:57.525 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:57.526 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 85 k 2024-03-29T11:46:57.526 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 231 k 2024-03-29T11:46:57.526 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-29T11:46:57.534 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:46:57.564 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:46:58.067 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 158 kB/s | 85 kB 00:00 2024-03-29T11:46:58.067 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:46:58.067 INFO:teuthology.orchestra.run.smithi121.stdout:Total 158 kB/s | 85 kB 00:00 2024-03-29T11:46:58.068 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-29T11:46:58.074 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-29T11:46:58.074 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-29T11:46:58.130 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-29T11:46:58.130 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-29T11:46:58.270 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-29T11:46:58.381 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:46:58.809 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:46:59.115 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:46:59.116 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:59.116 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-29T11:46:59.116 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:46:59.116 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:46:59.116 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:46:59.342 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-mirror 2024-03-29T11:46:59.923 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:02 ago on Fri 29 Mar 2024 11:44:57 AM UTC. 2024-03-29T11:47:00.081 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:47:00.081 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:47:00.081 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-03-29T11:47:00.082 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:47:00.082 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-29T11:47:00.082 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:47:00.082 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:47:00.082 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-29T11:47:00.082 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:47:00.082 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-03-29T11:47:00.083 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:47:00.083 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 3.1 M 2024-03-29T11:47:00.083 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 13 M 2024-03-29T11:47:00.083 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-29T11:47:00.167 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:47:00.167 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:47:00.167 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:47:00.167 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:47:00.167 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:47:00.168 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:47:00.168 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:47:00.168 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:47:00.168 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:47:00.168 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:47:00.169 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:47:00.169 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:47:00.642 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:47:00.643 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:00.643 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-29T11:47:00.643 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:47:00.643 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:47:00.644 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:47:00.644 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:47:00.644 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:47:00.645 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:47:00.645 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:47:00.645 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:47:00.645 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:47:00.645 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:47:00.645 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:47:00.645 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:47:00.647 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:00.648 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T11:47:01.062 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:47:01.176 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 2.8 MB/s | 3.1 MB 00:01 2024-03-29T11:47:01.177 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:47:01.177 INFO:teuthology.orchestra.run.smithi121.stdout:Total 2.8 MB/s | 3.1 MB 00:01 2024-03-29T11:47:01.177 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-29T11:47:01.187 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-29T11:47:01.187 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-29T11:47:01.263 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-29T11:47:01.263 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-29T11:47:01.632 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:37 ago on Fri 29 Mar 2024 11:45:24 AM UTC. 2024-03-29T11:47:01.710 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-29T11:47:01.760 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:47:01.783 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T11:47:01.784 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================== 2024-03-29T11:47:01.785 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-29T11:47:01.785 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================== 2024-03-29T11:47:01.785 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-29T11:47:01.785 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:47:01.785 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-29T11:47:01.785 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:47:01.785 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:47:01.785 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:47:01.785 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:47:01.785 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:47:01.785 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:47:01.785 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:47:01.785 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:47:01.786 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:47:01.786 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:47:01.786 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:47:01.786 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:47:01.786 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:01.786 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-29T11:47:01.786 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================== 2024-03-29T11:47:01.786 INFO:teuthology.orchestra.run.smithi098.stdout:Install 13 Packages 2024-03-29T11:47:01.786 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:01.787 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:47:01.787 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:47:01.787 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:47:01.787 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:47:01.787 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:47:01.787 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:47:01.787 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 2.0 M 2024-03-29T11:47:01.788 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 26 M 2024-03-29T11:47:01.788 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-29T11:47:02.265 INFO:teuthology.orchestra.run.smithi098.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 222 kB/s | 24 kB 00:00 2024-03-29T11:47:02.315 INFO:teuthology.orchestra.run.smithi098.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 113 kB/s | 18 kB 00:00 2024-03-29T11:47:02.357 INFO:teuthology.orchestra.run.smithi098.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.5 MB/s | 174 kB 00:00 2024-03-29T11:47:02.383 INFO:teuthology.orchestra.run.smithi098.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-29T11:47:02.410 INFO:teuthology.orchestra.run.smithi098.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.9 MB/s | 211 kB 00:00 2024-03-29T11:47:02.435 INFO:teuthology.orchestra.run.smithi098.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 832 kB/s | 43 kB 00:00 2024-03-29T11:47:02.544 INFO:teuthology.orchestra.run.smithi098.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 241 kB/s | 32 kB 00:00 2024-03-29T11:47:02.569 INFO:teuthology.orchestra.run.smithi098.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 112 kB/s | 15 kB 00:00 2024-03-29T11:47:02.602 INFO:teuthology.orchestra.run.smithi098.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 3.4 MB/s | 203 kB 00:00 2024-03-29T11:47:02.670 INFO:teuthology.orchestra.run.smithi098.stdout:(10/13): python3-kubernetes-26.1.0-2.el9.noarch 10 MB/s | 1.0 MB 00:00 2024-03-29T11:47:02.695 INFO:teuthology.orchestra.run.smithi098.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 645 kB/s | 59 kB 00:00 2024-03-29T11:47:02.720 INFO:teuthology.orchestra.run.smithi098.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-03-29T11:47:02.762 INFO:teuthology.orchestra.run.smithi098.stdout:(13/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el 86 kB/s | 52 kB 00:00 2024-03-29T11:47:02.765 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:47:02.765 INFO:teuthology.orchestra.run.smithi098.stdout:Total 2.1 MB/s | 2.0 MB 00:00 2024-03-29T11:47:02.765 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-29T11:47:02.778 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-29T11:47:02.778 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-29T11:47:02.866 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-29T11:47:02.866 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-29T11:47:02.931 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:47:02.931 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:47:02.931 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-29T11:47:02.931 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:47:02.931 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:47:02.931 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:47:03.166 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-nbd 2024-03-29T11:47:03.376 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-29T11:47:03.548 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:47:03.620 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:47:03.675 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:47:03.705 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:47:03.738 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:06 ago on Fri 29 Mar 2024 11:44:57 AM UTC. 2024-03-29T11:47:03.746 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:47:03.848 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:47:03.886 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:47:03.886 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:47:03.887 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-03-29T11:47:03.887 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:47:03.887 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-29T11:47:03.887 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:47:03.887 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:47:03.887 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-29T11:47:03.887 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:47:03.887 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-03-29T11:47:03.887 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:47:03.888 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 171 k 2024-03-29T11:47:03.888 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 494 k 2024-03-29T11:47:03.888 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-29T11:47:03.957 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:47:04.007 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:47:04.488 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 286 kB/s | 171 kB 00:00 2024-03-29T11:47:04.488 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:47:04.489 INFO:teuthology.orchestra.run.smithi121.stdout:Total 285 kB/s | 171 kB 00:00 2024-03-29T11:47:04.489 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-29T11:47:04.496 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-29T11:47:04.496 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-29T11:47:04.552 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-29T11:47:04.552 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-29T11:47:04.719 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:47:04.720 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-29T11:47:04.762 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:47:04.796 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:47:04.819 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:47:04.844 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:47:04.866 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:47:05.244 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:47:05.521 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:47:05.521 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:47:05.522 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-29T11:47:05.522 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:47:05.522 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:47:05.522 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:47:05.680 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:47:05.680 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:47:05.680 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:47:05.680 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:47:05.680 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:47:05.680 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:47:05.680 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:47:05.680 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:47:05.680 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:47:05.680 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:47:05.681 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:47:05.681 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:47:05.681 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:47:05.757 DEBUG:teuthology.parallel:result is None 2024-03-29T11:47:06.047 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:47:06.047 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:06.048 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-29T11:47:06.048 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:47:06.048 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:47:06.048 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:47:06.048 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:47:06.048 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:47:06.048 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:47:06.048 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:47:06.048 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:47:06.048 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:47:06.048 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:47:06.048 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:47:06.048 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:47:06.048 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:47:06.049 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:06.049 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T11:47:06.309 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:47:06.869 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 11:45:24 AM UTC. 2024-03-29T11:47:07.023 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T11:47:07.023 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================== 2024-03-29T11:47:07.023 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-29T11:47:07.024 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================== 2024-03-29T11:47:07.024 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-29T11:47:07.024 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:47:07.024 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-29T11:47:07.024 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:47:07.024 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:47:07.024 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:47:07.024 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:47:07.024 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:47:07.025 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:07.025 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-29T11:47:07.025 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================== 2024-03-29T11:47:07.025 INFO:teuthology.orchestra.run.smithi098.stdout:Install 6 Packages 2024-03-29T11:47:07.025 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:07.025 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 6.8 M 2024-03-29T11:47:07.025 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 33 M 2024-03-29T11:47:07.025 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-29T11:47:07.698 INFO:teuthology.orchestra.run.smithi098.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.5 MB/s | 248 kB 00:00 2024-03-29T11:47:08.123 INFO:teuthology.orchestra.run.smithi098.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 9.9 MB/s | 5.8 MB 00:00 2024-03-29T11:47:08.223 INFO:teuthology.orchestra.run.smithi098.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 201 kB/s | 138 kB 00:00 2024-03-29T11:47:08.257 INFO:teuthology.orchestra.run.smithi098.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 981 kB/s | 548 kB 00:00 2024-03-29T11:47:08.457 INFO:teuthology.orchestra.run.smithi098.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 173 kB/s | 58 kB 00:00 2024-03-29T11:47:08.515 INFO:teuthology.orchestra.run.smithi098.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 175 kB/s | 51 kB 00:00 2024-03-29T11:47:08.517 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:47:08.517 INFO:teuthology.orchestra.run.smithi098.stdout:Total 4.5 MB/s | 6.8 MB 00:01 2024-03-29T11:47:08.531 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-29T11:47:08.541 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-29T11:47:08.541 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-29T11:47:08.628 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-29T11:47:08.629 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-29T11:47:08.933 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-29T11:47:09.091 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:47:09.142 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:47:09.654 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:47:09.745 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:47:09.807 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:47:09.830 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:47:10.659 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:47:10.660 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:47:10.660 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:47:10.660 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:47:10.660 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:47:10.660 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:47:10.981 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:47:10.981 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:10.981 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-29T11:47:10.981 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:47:10.982 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:47:10.982 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:47:10.982 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:47:10.982 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:47:10.982 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:47:10.982 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:10.982 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T11:47:11.234 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-fuse 2024-03-29T11:47:11.795 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 11:45:24 AM UTC. 2024-03-29T11:47:11.945 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T11:47:11.945 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T11:47:11.946 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-29T11:47:11.946 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T11:47:11.946 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-29T11:47:11.946 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:47:11.946 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-29T11:47:11.946 INFO:teuthology.orchestra.run.smithi098.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:47:11.946 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:11.946 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-29T11:47:11.946 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T11:47:11.946 INFO:teuthology.orchestra.run.smithi098.stdout:Install 2 Packages 2024-03-29T11:47:11.946 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:11.946 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 964 k 2024-03-29T11:47:11.946 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 2.8 M 2024-03-29T11:47:11.947 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-29T11:47:12.061 INFO:teuthology.orchestra.run.smithi098.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 690 kB/s | 79 kB 00:00 2024-03-29T11:47:13.169 INFO:teuthology.orchestra.run.smithi098.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 724 kB/s | 885 kB 00:01 2024-03-29T11:47:13.170 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:47:13.171 INFO:teuthology.orchestra.run.smithi098.stdout:Total 787 kB/s | 964 kB 00:01 2024-03-29T11:47:13.171 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-29T11:47:13.192 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-29T11:47:13.192 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-29T11:47:13.266 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-29T11:47:13.267 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-29T11:47:13.450 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-29T11:47:13.595 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:47:13.661 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:47:14.334 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:47:14.334 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:47:14.647 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:47:14.647 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:14.647 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-29T11:47:14.647 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:47:14.648 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:14.648 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T11:47:14.902 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-volume 2024-03-29T11:47:15.480 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:51 ago on Fri 29 Mar 2024 11:45:24 AM UTC. 2024-03-29T11:47:15.636 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T11:47:15.636 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T11:47:15.636 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-29T11:47:15.636 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T11:47:15.637 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-29T11:47:15.637 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:47:15.637 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-29T11:47:15.637 INFO:teuthology.orchestra.run.smithi098.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:47:15.637 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:15.637 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-29T11:47:15.637 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T11:47:15.637 INFO:teuthology.orchestra.run.smithi098.stdout:Install 2 Packages 2024-03-29T11:47:15.637 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:15.637 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 332 k 2024-03-29T11:47:15.637 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 1.5 M 2024-03-29T11:47:15.637 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-29T11:47:15.749 INFO:teuthology.orchestra.run.smithi098.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 618 kB/s | 69 kB 00:00 2024-03-29T11:47:16.541 INFO:teuthology.orchestra.run.smithi098.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 292 kB/s | 264 kB 00:00 2024-03-29T11:47:16.542 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:47:16.543 INFO:teuthology.orchestra.run.smithi098.stdout:Total 367 kB/s | 332 kB 00:00 2024-03-29T11:47:16.543 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-29T11:47:16.552 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-29T11:47:16.552 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-29T11:47:16.591 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-29T11:47:16.591 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-29T11:47:16.743 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-29T11:47:16.949 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:47:16.985 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:47:17.002 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:47:17.003 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:47:17.003 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:47:17.003 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:17.711 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:47:18.053 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:47:18.053 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:18.054 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-29T11:47:18.054 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:47:18.054 INFO:teuthology.orchestra.run.smithi098.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:47:18.054 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:18.054 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T11:47:18.380 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install librados-devel 2024-03-29T11:47:18.950 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:54 ago on Fri 29 Mar 2024 11:45:24 AM UTC. 2024-03-29T11:47:19.101 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T11:47:19.102 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T11:47:19.102 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-29T11:47:19.103 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T11:47:19.103 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-29T11:47:19.103 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:47:19.103 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:19.103 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-29T11:47:19.103 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T11:47:19.103 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-29T11:47:19.103 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:19.103 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 127 k 2024-03-29T11:47:19.103 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 456 k 2024-03-29T11:47:19.103 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-29T11:47:19.573 INFO:teuthology.orchestra.run.smithi098.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 270 kB/s | 127 kB 00:00 2024-03-29T11:47:19.574 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:47:19.574 INFO:teuthology.orchestra.run.smithi098.stdout:Total 269 kB/s | 127 kB 00:00 2024-03-29T11:47:19.574 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-29T11:47:19.579 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-29T11:47:19.580 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-29T11:47:19.636 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-29T11:47:19.636 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-29T11:47:19.811 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-29T11:47:19.914 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:47:20.371 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:47:20.677 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:47:20.677 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:20.677 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-29T11:47:20.677 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:47:20.677 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:20.677 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T11:47:20.935 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install libcephfs2 2024-03-29T11:47:21.513 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:57 ago on Fri 29 Mar 2024 11:45:24 AM UTC. 2024-03-29T11:47:21.614 INFO:teuthology.orchestra.run.smithi098.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:47:21.668 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T11:47:21.669 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-03-29T11:47:21.669 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T11:47:21.752 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install libcephfs-devel 2024-03-29T11:47:22.316 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:58 ago on Fri 29 Mar 2024 11:45:24 AM UTC. 2024-03-29T11:47:22.466 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T11:47:22.467 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T11:47:22.467 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-29T11:47:22.468 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T11:47:22.468 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-29T11:47:22.468 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:47:22.468 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:22.468 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-29T11:47:22.468 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T11:47:22.468 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-29T11:47:22.468 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:22.468 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 31 k 2024-03-29T11:47:22.468 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 140 k 2024-03-29T11:47:22.468 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-29T11:47:22.643 INFO:teuthology.orchestra.run.smithi098.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 178 kB/s | 31 kB 00:00 2024-03-29T11:47:22.644 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:47:22.644 INFO:teuthology.orchestra.run.smithi098.stdout:Total 176 kB/s | 31 kB 00:00 2024-03-29T11:47:22.644 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-29T11:47:22.648 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-29T11:47:22.648 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-29T11:47:22.674 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-29T11:47:22.675 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-29T11:47:22.807 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-29T11:47:22.900 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:47:23.267 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:47:23.579 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:47:23.579 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:23.579 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-29T11:47:23.580 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:47:23.580 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:23.581 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T11:47:23.761 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install librados2 2024-03-29T11:47:24.324 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:02:00 ago on Fri 29 Mar 2024 11:45:24 AM UTC. 2024-03-29T11:47:24.422 INFO:teuthology.orchestra.run.smithi098.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:47:24.474 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T11:47:24.478 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-03-29T11:47:24.478 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T11:47:24.532 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install librbd1 2024-03-29T11:47:25.090 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:02:01 ago on Fri 29 Mar 2024 11:45:24 AM UTC. 2024-03-29T11:47:25.187 INFO:teuthology.orchestra.run.smithi098.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:47:25.238 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T11:47:25.239 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-03-29T11:47:25.239 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T11:47:25.284 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-rados 2024-03-29T11:47:25.849 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:02:01 ago on Fri 29 Mar 2024 11:45:24 AM UTC. 2024-03-29T11:47:25.948 INFO:teuthology.orchestra.run.smithi098.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:47:25.999 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T11:47:26.000 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-03-29T11:47:26.000 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T11:47:26.055 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-rgw 2024-03-29T11:47:26.614 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:02:02 ago on Fri 29 Mar 2024 11:45:24 AM UTC. 2024-03-29T11:47:26.712 INFO:teuthology.orchestra.run.smithi098.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:47:26.764 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T11:47:26.765 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-03-29T11:47:26.765 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T11:47:26.822 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-cephfs 2024-03-29T11:47:27.430 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:02:03 ago on Fri 29 Mar 2024 11:45:24 AM UTC. 2024-03-29T11:47:27.528 INFO:teuthology.orchestra.run.smithi098.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:47:27.579 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T11:47:27.580 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-03-29T11:47:27.580 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T11:47:27.630 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-rbd 2024-03-29T11:47:28.195 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:02:04 ago on Fri 29 Mar 2024 11:45:24 AM UTC. 2024-03-29T11:47:28.294 INFO:teuthology.orchestra.run.smithi098.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:47:28.346 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T11:47:28.347 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-03-29T11:47:28.347 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T11:47:28.397 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install rbd-fuse 2024-03-29T11:47:28.954 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:02:04 ago on Fri 29 Mar 2024 11:45:24 AM UTC. 2024-03-29T11:47:29.101 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T11:47:29.101 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T11:47:29.102 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-03-29T11:47:29.102 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T11:47:29.102 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-29T11:47:29.102 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:47:29.102 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:29.102 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-29T11:47:29.102 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T11:47:29.102 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-29T11:47:29.103 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:29.103 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 85 k 2024-03-29T11:47:29.103 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 231 k 2024-03-29T11:47:29.103 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-29T11:47:29.313 INFO:teuthology.orchestra.run.smithi098.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 403 kB/s | 85 kB 00:00 2024-03-29T11:47:29.314 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:47:29.314 INFO:teuthology.orchestra.run.smithi098.stdout:Total 401 kB/s | 85 kB 00:00 2024-03-29T11:47:29.314 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-29T11:47:29.321 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-29T11:47:29.322 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-29T11:47:29.377 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-29T11:47:29.377 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-29T11:47:29.512 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-29T11:47:29.621 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:47:30.005 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:47:30.301 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:47:30.302 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:30.302 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-29T11:47:30.302 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:47:30.302 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:30.302 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T11:47:30.531 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install rbd-mirror 2024-03-29T11:47:31.103 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:02:07 ago on Fri 29 Mar 2024 11:45:24 AM UTC. 2024-03-29T11:47:31.257 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T11:47:31.258 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T11:47:31.258 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-29T11:47:31.258 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T11:47:31.259 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-29T11:47:31.259 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:47:31.259 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:31.259 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-29T11:47:31.259 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T11:47:31.259 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-29T11:47:31.259 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:31.259 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 3.1 M 2024-03-29T11:47:31.259 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 13 M 2024-03-29T11:47:31.259 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-29T11:47:31.659 INFO:teuthology.orchestra.run.smithi098.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 7.7 MB/s | 3.1 MB 00:00 2024-03-29T11:47:31.659 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:47:31.660 INFO:teuthology.orchestra.run.smithi098.stdout:Total 7.7 MB/s | 3.1 MB 00:00 2024-03-29T11:47:31.660 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-29T11:47:31.669 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-29T11:47:31.670 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-29T11:47:31.742 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-29T11:47:31.742 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-29T11:47:32.174 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-29T11:47:32.244 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:47:32.268 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:47:32.268 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:47:32.268 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:47:32.269 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:47:32.269 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:47:32.269 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:33.394 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:47:33.394 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:33.394 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-29T11:47:33.394 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:47:33.394 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:33.394 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T11:47:33.641 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install rbd-nbd 2024-03-29T11:47:34.218 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:02:10 ago on Fri 29 Mar 2024 11:45:24 AM UTC. 2024-03-29T11:47:34.366 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T11:47:34.366 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T11:47:34.367 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-03-29T11:47:34.367 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T11:47:34.367 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-29T11:47:34.367 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:47:34.367 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:34.367 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-29T11:47:34.367 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T11:47:34.367 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-29T11:47:34.368 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:34.368 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 171 k 2024-03-29T11:47:34.368 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 494 k 2024-03-29T11:47:34.368 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-29T11:47:34.713 INFO:teuthology.orchestra.run.smithi098.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 497 kB/s | 171 kB 00:00 2024-03-29T11:47:34.714 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:47:34.714 INFO:teuthology.orchestra.run.smithi098.stdout:Total 495 kB/s | 171 kB 00:00 2024-03-29T11:47:34.714 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-29T11:47:34.722 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-29T11:47:34.722 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-29T11:47:34.780 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-29T11:47:34.781 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-29T11:47:34.926 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-29T11:47:35.036 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:47:35.467 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:47:35.777 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:47:35.777 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:35.778 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-29T11:47:35.778 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:47:35.778 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:35.778 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T11:47:36.010 DEBUG:teuthology.parallel:result is None 2024-03-29T11:47:36.010 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:47:36.167 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:47:36.191 INFO:teuthology.orchestra.run.smithi098.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:47:36.192 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:47:36.192 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:47:36.194 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:47:36.423 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:47:36.448 INFO:teuthology.orchestra.run.smithi121.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:47:36.449 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:47:36.449 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:47:36.451 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T11:47:36.451 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T11:47:36.451 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:47:36.480 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-29T11:47:36.480 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:47:36.520 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T11:47:36.546 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T11:47:36.546 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:47:36.610 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:47:36.678 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-29T11:47:36.678 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:47:36.708 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:47:36.776 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T11:47:36.776 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T11:47:36.776 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:47:36.803 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:47:36.870 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-29T11:47:36.871 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:47:36.898 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:47:36.965 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T11:47:36.966 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T11:47:36.966 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:47:36.992 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:47:37.058 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-29T11:47:37.058 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:47:37.091 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:47:37.158 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T11:47:37.382 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T11:47:37.382 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 777 /var/log/ceph 2024-03-29T11:47:37.384 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 777 /var/log/ceph 2024-03-29T11:47:37.411 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T11:47:37.411 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:47:37.452 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:47:37.479 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T11:47:37.479 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:47:37.521 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:47:37.590 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T11:47:37.590 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T11:47:37.590 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622735', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622735', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi098.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFYgWZ1LbgwbHhpk7VTsXP9V/QXU8ImHq/fUj86pKZUcizIBTgap0pTq4nwuQdS5UBBTwAE0Tft+80B3oG7NyjA=', 'smithi121.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFEeDHpd8zmJPFCJ6dqE7CuXyzXFk53/XMEvbQKq08tzZzHUQC4pkdpPPyFRqyFEUI2I65wyAYFYYTc38KvJnzE='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T11:47:37.591 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:47:37.594 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:47:37.608 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:47:37.648 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:47:37.675 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T11:47:37.676 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:47:37.729 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:47:37.729 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_1 2024-03-29T11:47:37.783 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:47:37.783 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:47:37.784 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-29T11:47:37.784 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:47:37.784 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:47:37.784 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-29 11:47:32.513979401 +0000 2024-03-29T11:47:37.784 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-29 11:44:11.285316018 +0000 2024-03-29T11:47:37.784 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-29 11:44:11.285316018 +0000 2024-03-29T11:47:37.784 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-03-29 11:44:11.285316018 +0000 2024-03-29T11:47:37.784 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:47:37.849 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-29T11:47:37.849 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-29T11:47:37.850 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.00020862 s, 2.5 MB/s 2024-03-29T11:47:37.851 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:47:37.907 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_2 2024-03-29T11:47:37.962 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:47:37.962 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:47:37.962 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-29T11:47:37.962 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:47:37.962 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:47:37.963 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-29 11:47:32.513979401 +0000 2024-03-29T11:47:37.963 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-29 11:44:11.629311589 +0000 2024-03-29T11:47:37.963 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-29 11:44:11.629311589 +0000 2024-03-29T11:47:37.963 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-03-29 11:44:11.629311589 +0000 2024-03-29T11:47:37.963 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:47:38.028 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-29T11:47:38.028 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-29T11:47:38.028 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000207021 s, 2.5 MB/s 2024-03-29T11:47:38.029 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:47:38.092 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_3 2024-03-29T11:47:38.147 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:47:38.147 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:47:38.147 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-29T11:47:38.147 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:47:38.147 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:47:38.147 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-29 11:47:32.513979401 +0000 2024-03-29T11:47:38.147 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-29 11:44:11.976307121 +0000 2024-03-29T11:47:38.147 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-29 11:44:11.976307121 +0000 2024-03-29T11:47:38.147 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-03-29 11:44:11.976307121 +0000 2024-03-29T11:47:38.148 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:47:38.237 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-29T11:47:38.237 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-29T11:47:38.237 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000169492 s, 3.0 MB/s 2024-03-29T11:47:38.237 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:47:38.302 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_4 2024-03-29T11:47:38.365 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:47:38.365 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:47:38.365 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-29T11:47:38.365 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:47:38.365 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:47:38.365 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-29 11:47:32.514979389 +0000 2024-03-29T11:47:38.366 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-29 11:44:12.301302936 +0000 2024-03-29T11:47:38.366 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-29 11:44:12.301302936 +0000 2024-03-29T11:47:38.366 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-03-29 11:44:12.301302936 +0000 2024-03-29T11:47:38.366 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:47:38.436 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-29T11:47:38.436 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-29T11:47:38.436 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000208164 s, 2.5 MB/s 2024-03-29T11:47:38.437 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:47:38.502 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:47:38.502 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-29T11:47:38.502 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:47:38.520 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:47:38.520 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_1 2024-03-29T11:47:38.577 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:47:38.577 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:47:38.577 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-29T11:47:38.577 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:47:38.577 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:47:38.577 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-03-29 11:47:02.029916854 +0000 2024-03-29T11:47:38.577 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-03-29 11:44:11.293957563 +0000 2024-03-29T11:47:38.577 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-03-29 11:44:11.293957563 +0000 2024-03-29T11:47:38.578 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-03-29 11:44:11.293957563 +0000 2024-03-29T11:47:38.578 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:47:38.645 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-03-29T11:47:38.645 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-03-29T11:47:38.645 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000208604 s, 2.5 MB/s 2024-03-29T11:47:38.647 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:47:38.706 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_2 2024-03-29T11:47:38.763 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:47:38.763 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:47:38.763 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-29T11:47:38.763 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:47:38.763 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:47:38.763 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-03-29 11:47:02.029916854 +0000 2024-03-29T11:47:38.763 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-03-29 11:44:11.663952636 +0000 2024-03-29T11:47:38.763 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-03-29 11:44:11.663952636 +0000 2024-03-29T11:47:38.763 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-03-29 11:44:11.663952636 +0000 2024-03-29T11:47:38.764 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:47:38.830 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-03-29T11:47:38.831 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-03-29T11:47:38.831 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000232081 s, 2.2 MB/s 2024-03-29T11:47:38.832 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:47:38.891 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_3 2024-03-29T11:47:38.947 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:47:38.947 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:47:38.947 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-29T11:47:38.947 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:47:38.948 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:47:38.948 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-03-29 11:47:02.029916854 +0000 2024-03-29T11:47:38.948 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-03-29 11:44:12.007948056 +0000 2024-03-29T11:47:38.948 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-03-29 11:44:12.007948056 +0000 2024-03-29T11:47:38.948 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-03-29 11:44:12.007948056 +0000 2024-03-29T11:47:38.948 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:47:39.022 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-03-29T11:47:39.024 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-03-29T11:47:39.024 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000187033 s, 2.7 MB/s 2024-03-29T11:47:39.026 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:47:39.082 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_4 2024-03-29T11:47:39.145 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:47:39.145 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:47:39.146 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-29T11:47:39.146 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:47:39.146 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:47:39.146 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-03-29 11:47:02.030916841 +0000 2024-03-29T11:47:39.146 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-03-29 11:44:12.352943463 +0000 2024-03-29T11:47:39.146 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-03-29 11:44:12.352943463 +0000 2024-03-29T11:47:39.146 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-03-29 11:44:12.352943463 +0000 2024-03-29T11:47:39.146 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:47:39.222 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-03-29T11:47:39.222 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-03-29T11:47:39.222 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000244749 s, 2.1 MB/s 2024-03-29T11:47:39.223 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:47:39.281 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:47:39.281 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi098.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi121.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:47:39.281 INFO:tasks.ceph:Generating config... 2024-03-29T11:47:39.282 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T11:47:39.282 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T11:47:39.282 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T11:47:39.282 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T11:47:39.282 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T11:47:39.282 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T11:47:39.282 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T11:47:39.282 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T11:47:39.283 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T11:47:39.283 INFO:tasks.ceph:[global] ms type = async 2024-03-29T11:47:39.283 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T11:47:39.283 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T11:47:39.283 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T11:47:39.283 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T11:47:39.283 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T11:47:39.283 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T11:47:39.283 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T11:47:39.283 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T11:47:39.283 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T11:47:39.283 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T11:47:39.283 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T11:47:39.283 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T11:47:39.283 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T11:47:39.284 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-03-29T11:47:39.284 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T11:47:39.284 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T11:47:39.284 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T11:47:39.284 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T11:47:39.284 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T11:47:39.284 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T11:47:39.284 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T11:47:39.284 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T11:47:39.284 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T11:47:39.284 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T11:47:39.284 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T11:47:39.284 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T11:47:39.284 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T11:47:39.284 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T11:47:39.285 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T11:47:39.285 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T11:47:39.285 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T11:47:39.285 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T11:47:39.285 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T11:47:39.285 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T11:47:39.285 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T11:47:39.285 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T11:47:39.285 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T11:47:39.285 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T11:47:39.285 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T11:47:39.285 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T11:47:39.351 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T11:47:39.353 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T11:47:39.400 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:47:39.472 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.98:6789]'), ('mon.c', '[v1:172.21.15.98:6790]'), ('mon.b', '[v1:172.21.15.121:6789]')] 2024-03-29T11:47:39.472 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.98:6789],[v1:172.21.15.98:6790],[v1:172.21.15.121:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T11:47:39.473 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T11:47:39.473 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:47:39.534 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.98:6789]' --addv c '[v1:172.21.15.98:6790]' --addv b '[v1:172.21.15.121:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:47:39.629 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:47:39.629 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool: generated fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:47:39.630 INFO:teuthology.orchestra.run.smithi098.stdout:setting min_mon_release = quincy 2024-03-29T11:47:39.630 INFO:teuthology.orchestra.run.smithi098.stdout:epoch 0 2024-03-29T11:47:39.630 INFO:teuthology.orchestra.run.smithi098.stdout:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:47:39.630 INFO:teuthology.orchestra.run.smithi098.stdout:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:47:39.630 INFO:teuthology.orchestra.run.smithi098.stdout:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:47:39.630 INFO:teuthology.orchestra.run.smithi098.stdout:min_mon_release 17 (quincy) 2024-03-29T11:47:39.630 INFO:teuthology.orchestra.run.smithi098.stdout:election_strategy: 1 2024-03-29T11:47:39.631 INFO:teuthology.orchestra.run.smithi098.stdout:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:47:39.631 INFO:teuthology.orchestra.run.smithi098.stdout:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:47:39.631 INFO:teuthology.orchestra.run.smithi098.stdout:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:47:39.631 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T11:47:39.632 DEBUG:teuthology.orchestra.run.smithi098:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:47:39.653 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 683631e6-f6c8-470c-ab2f-82b5e5343501... 2024-03-29T11:47:39.654 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:47:39.698 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:47:39.746 INFO:teuthology.orchestra.run.smithi098.stdout:[global] 2024-03-29T11:47:39.746 INFO:teuthology.orchestra.run.smithi098.stdout: chdir = "" 2024-03-29T11:47:39.747 INFO:teuthology.orchestra.run.smithi098.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:47:39.747 INFO:teuthology.orchestra.run.smithi098.stdout: auth supported = cephx 2024-03-29T11:47:39.747 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:39.747 INFO:teuthology.orchestra.run.smithi098.stdout: filestore xattr use omap = true 2024-03-29T11:47:39.748 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:39.748 INFO:teuthology.orchestra.run.smithi098.stdout: mon clock drift allowed = 1.000 2024-03-29T11:47:39.748 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:39.748 INFO:teuthology.orchestra.run.smithi098.stdout: osd crush chooseleaf type = 0 2024-03-29T11:47:39.748 INFO:teuthology.orchestra.run.smithi098.stdout: auth debug = true 2024-03-29T11:47:39.748 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:39.748 INFO:teuthology.orchestra.run.smithi098.stdout: ms die on old message = true 2024-03-29T11:47:39.749 INFO:teuthology.orchestra.run.smithi098.stdout: ms die on bug = true 2024-03-29T11:47:39.749 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:39.749 INFO:teuthology.orchestra.run.smithi098.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:47:39.749 INFO:teuthology.orchestra.run.smithi098.stdout: mon pg warn max object skew = 0 2024-03-29T11:47:39.749 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:39.750 INFO:teuthology.orchestra.run.smithi098.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:47:39.750 INFO:teuthology.orchestra.run.smithi098.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:47:39.750 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:39.750 INFO:teuthology.orchestra.run.smithi098.stdout: osd pool default size = 2 2024-03-29T11:47:39.750 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:39.750 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd allow primary affinity = true 2024-03-29T11:47:39.751 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd allow pg remap = true 2024-03-29T11:47:39.751 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:47:39.751 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:47:39.751 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on no sortbitwise = false 2024-03-29T11:47:39.751 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:47:39.751 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on too few osds = false 2024-03-29T11:47:39.751 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:47:39.751 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:47:39.751 INFO:teuthology.orchestra.run.smithi098.stdout: mon_allow_pool_size_one = true 2024-03-29T11:47:39.751 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:39.751 INFO:teuthology.orchestra.run.smithi098.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:47:39.751 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:39.751 INFO:teuthology.orchestra.run.smithi098.stdout: osd default data pool replay window = 5 2024-03-29T11:47:39.751 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:39.751 INFO:teuthology.orchestra.run.smithi098.stdout: mon allow pool delete = true 2024-03-29T11:47:39.752 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:39.752 INFO:teuthology.orchestra.run.smithi098.stdout: mon cluster log file level = debug 2024-03-29T11:47:39.752 INFO:teuthology.orchestra.run.smithi098.stdout: debug asserts on shutdown = true 2024-03-29T11:47:39.752 INFO:teuthology.orchestra.run.smithi098.stdout: mon health detail to clog = false 2024-03-29T11:47:39.752 INFO:teuthology.orchestra.run.smithi098.stdout: mon host = "[v1:172.21.15.98:6789],[v1:172.21.15.98:6790],[v1:172.21.15.121:6789]" 2024-03-29T11:47:39.752 INFO:teuthology.orchestra.run.smithi098.stdout: lockdep = True 2024-03-29T11:47:39.752 INFO:teuthology.orchestra.run.smithi098.stdout: mon client directed command retry = 5 2024-03-29T11:47:39.752 INFO:teuthology.orchestra.run.smithi098.stdout: mon election default strategy = 1 2024-03-29T11:47:39.752 INFO:teuthology.orchestra.run.smithi098.stdout: ms bind msgr2 = False 2024-03-29T11:47:39.752 INFO:teuthology.orchestra.run.smithi098.stdout: ms inject socket failures = 5000 2024-03-29T11:47:39.752 INFO:teuthology.orchestra.run.smithi098.stdout: ms type = async 2024-03-29T11:47:39.752 INFO:teuthology.orchestra.run.smithi098.stdout: fsid = 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:47:39.752 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:39.752 INFO:teuthology.orchestra.run.smithi098.stdout:[osd] 2024-03-29T11:47:39.752 INFO:teuthology.orchestra.run.smithi098.stdout: osd journal size = 100 2024-03-29T11:47:39.753 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:39.753 INFO:teuthology.orchestra.run.smithi098.stdout: osd scrub load threshold = 5.0 2024-03-29T11:47:39.753 INFO:teuthology.orchestra.run.smithi098.stdout: osd scrub max interval = 600 2024-03-29T11:47:39.753 INFO:teuthology.orchestra.run.smithi098.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:47:39.753 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:39.753 INFO:teuthology.orchestra.run.smithi098.stdout: osd recover clone overlap = true 2024-03-29T11:47:39.753 INFO:teuthology.orchestra.run.smithi098.stdout: osd recovery max chunk = 1048576 2024-03-29T11:47:39.754 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:39.754 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug shutdown = true 2024-03-29T11:47:39.754 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug op order = true 2024-03-29T11:47:39.754 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug verify stray on activate = true 2024-03-29T11:47:39.754 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug trim objects = true 2024-03-29T11:47:39.754 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:39.754 INFO:teuthology.orchestra.run.smithi098.stdout: osd open classes on start = true 2024-03-29T11:47:39.754 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug pg log writeout = true 2024-03-29T11:47:39.754 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:39.754 INFO:teuthology.orchestra.run.smithi098.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:47:39.755 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:39.755 INFO:teuthology.orchestra.run.smithi098.stdout: osd map max advance = 10 2024-03-29T11:47:39.755 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:39.755 INFO:teuthology.orchestra.run.smithi098.stdout: journal zero on create = true 2024-03-29T11:47:39.755 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:39.755 INFO:teuthology.orchestra.run.smithi098.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:47:39.755 INFO:teuthology.orchestra.run.smithi098.stdout: filestore apply finisher threads = 3 2024-03-29T11:47:39.755 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:39.755 INFO:teuthology.orchestra.run.smithi098.stdout: bdev debug aio = true 2024-03-29T11:47:39.755 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug misdirected ops = true 2024-03-29T11:47:39.755 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore block size = 96636764160 2024-03-29T11:47:39.755 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore compression algorithm = snappy 2024-03-29T11:47:39.755 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore compression mode = aggressive 2024-03-29T11:47:39.756 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore fsck on mount = True 2024-03-29T11:47:39.756 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore zero block detection = True 2024-03-29T11:47:39.756 INFO:teuthology.orchestra.run.smithi098.stdout: debug bluefs = 20 2024-03-29T11:47:39.756 INFO:teuthology.orchestra.run.smithi098.stdout: debug bluestore = 20 2024-03-29T11:47:39.756 INFO:teuthology.orchestra.run.smithi098.stdout: debug monc = 20 2024-03-29T11:47:39.756 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-03-29T11:47:39.756 INFO:teuthology.orchestra.run.smithi098.stdout: debug osd = 20 2024-03-29T11:47:39.756 INFO:teuthology.orchestra.run.smithi098.stdout: debug rocksdb = 10 2024-03-29T11:47:39.756 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:47:39.756 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd full ratio = 0.9 2024-03-29T11:47:39.756 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:47:39.756 INFO:teuthology.orchestra.run.smithi098.stdout: osd class default list = * 2024-03-29T11:47:39.756 INFO:teuthology.orchestra.run.smithi098.stdout: osd class load list = * 2024-03-29T11:47:39.756 INFO:teuthology.orchestra.run.smithi098.stdout: osd client watch timeout = 120 2024-03-29T11:47:39.756 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug verify cached snaps = True 2024-03-29T11:47:39.757 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug verify missing on start = True 2024-03-29T11:47:39.757 INFO:teuthology.orchestra.run.smithi098.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:47:39.757 INFO:teuthology.orchestra.run.smithi098.stdout: osd mclock override recovery settings = True 2024-03-29T11:47:39.757 INFO:teuthology.orchestra.run.smithi098.stdout: osd objectstore = bluestore 2024-03-29T11:47:39.757 INFO:teuthology.orchestra.run.smithi098.stdout: osd op queue = debug_random 2024-03-29T11:47:39.757 INFO:teuthology.orchestra.run.smithi098.stdout: osd op queue cut off = debug_random 2024-03-29T11:47:39.757 INFO:teuthology.orchestra.run.smithi098.stdout: osd shutdown pgref assert = True 2024-03-29T11:47:39.757 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:39.757 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr] 2024-03-29T11:47:39.757 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-03-29T11:47:39.757 INFO:teuthology.orchestra.run.smithi098.stdout: debug mgr = 20 2024-03-29T11:47:39.757 INFO:teuthology.orchestra.run.smithi098.stdout: debug mon = 20 2024-03-29T11:47:39.757 INFO:teuthology.orchestra.run.smithi098.stdout: debug auth = 20 2024-03-29T11:47:39.757 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:47:39.757 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:47:39.757 INFO:teuthology.orchestra.run.smithi098.stdout: mgr/telemetry/nag = false 2024-03-29T11:47:39.758 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:39.758 INFO:teuthology.orchestra.run.smithi098.stdout:[mon] 2024-03-29T11:47:39.758 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-03-29T11:47:39.758 INFO:teuthology.orchestra.run.smithi098.stdout: debug mon = 20 2024-03-29T11:47:39.758 INFO:teuthology.orchestra.run.smithi098.stdout: debug paxos = 20 2024-03-29T11:47:39.758 INFO:teuthology.orchestra.run.smithi098.stdout: debug auth = 20 2024-03-29T11:47:39.758 INFO:teuthology.orchestra.run.smithi098.stdout: mon data avail warn = 5 2024-03-29T11:47:39.759 INFO:teuthology.orchestra.run.smithi098.stdout: mon mgr mkfs grace = 240 2024-03-29T11:47:39.759 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:47:39.759 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd reporter subtree level = osd 2024-03-29T11:47:39.759 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd prime pg temp = true 2024-03-29T11:47:39.759 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:47:39.759 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:39.759 INFO:teuthology.orchestra.run.smithi098.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:47:39.759 INFO:teuthology.orchestra.run.smithi098.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:47:39.759 INFO:teuthology.orchestra.run.smithi098.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:47:39.759 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:39.759 INFO:teuthology.orchestra.run.smithi098.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:47:39.759 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:47:39.759 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:47:39.760 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:39.760 INFO:teuthology.orchestra.run.smithi098.stdout: # 1m isn't quite enough 2024-03-29T11:47:39.760 INFO:teuthology.orchestra.run.smithi098.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:47:39.760 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:39.760 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:47:39.760 INFO:teuthology.orchestra.run.smithi098.stdout: mon min osdmap epochs = 50 2024-03-29T11:47:39.760 INFO:teuthology.orchestra.run.smithi098.stdout: mon osdmap full prune interval = 2 2024-03-29T11:47:39.760 INFO:teuthology.orchestra.run.smithi098.stdout: mon osdmap full prune min = 15 2024-03-29T11:47:39.760 INFO:teuthology.orchestra.run.smithi098.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:47:39.760 INFO:teuthology.orchestra.run.smithi098.stdout: mon scrub interval = 300 2024-03-29T11:47:39.760 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on pool no app = False 2024-03-29T11:47:39.760 INFO:teuthology.orchestra.run.smithi098.stdout: paxos service trim min = 10 2024-03-29T11:47:39.760 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:47:39.760 INFO:teuthology.orchestra.run.smithi098.stdout:[client] 2024-03-29T11:47:39.760 INFO:teuthology.orchestra.run.smithi098.stdout: rgw cache enabled = true 2024-03-29T11:47:39.761 INFO:teuthology.orchestra.run.smithi098.stdout: rgw enable ops log = true 2024-03-29T11:47:39.761 INFO:teuthology.orchestra.run.smithi098.stdout: rgw enable usage log = true 2024-03-29T11:47:39.761 INFO:teuthology.orchestra.run.smithi098.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:47:39.761 INFO:teuthology.orchestra.run.smithi098.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:47:39.761 INFO:teuthology.orchestra.run.smithi098.stdout: debug monc = 20 2024-03-29T11:47:39.761 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-03-29T11:47:39.761 INFO:teuthology.orchestra.run.smithi098.stdout: debug objecter = 20 2024-03-29T11:47:39.761 INFO:teuthology.orchestra.run.smithi098.stdout: debug rados = 20 2024-03-29T11:47:39.761 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.a] 2024-03-29T11:47:39.761 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.c] 2024-03-29T11:47:39.761 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.b] 2024-03-29T11:47:39.762 INFO:teuthology.orchestra.run.smithi121.stdout:[global] 2024-03-29T11:47:39.762 INFO:teuthology.orchestra.run.smithi121.stdout: chdir = "" 2024-03-29T11:47:39.762 INFO:teuthology.orchestra.run.smithi121.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:47:39.762 INFO:teuthology.orchestra.run.smithi121.stdout: auth supported = cephx 2024-03-29T11:47:39.762 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:47:39.762 INFO:teuthology.orchestra.run.smithi121.stdout: filestore xattr use omap = true 2024-03-29T11:47:39.762 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:47:39.762 INFO:teuthology.orchestra.run.smithi121.stdout: mon clock drift allowed = 1.000 2024-03-29T11:47:39.763 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:47:39.763 INFO:teuthology.orchestra.run.smithi121.stdout: osd crush chooseleaf type = 0 2024-03-29T11:47:39.763 INFO:teuthology.orchestra.run.smithi121.stdout: auth debug = true 2024-03-29T11:47:39.763 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:47:39.763 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on old message = true 2024-03-29T11:47:39.763 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on bug = true 2024-03-29T11:47:39.763 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:47:39.763 INFO:teuthology.orchestra.run.smithi121.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:47:39.763 INFO:teuthology.orchestra.run.smithi121.stdout: mon pg warn max object skew = 0 2024-03-29T11:47:39.763 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:47:39.763 INFO:teuthology.orchestra.run.smithi121.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:47:39.763 INFO:teuthology.orchestra.run.smithi121.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:47:39.764 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:47:39.764 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default size = 2 2024-03-29T11:47:39.764 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:47:39.765 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow primary affinity = true 2024-03-29T11:47:39.765 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow pg remap = true 2024-03-29T11:47:39.765 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:47:39.766 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:47:39.766 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on no sortbitwise = false 2024-03-29T11:47:39.766 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:47:39.766 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on too few osds = false 2024-03-29T11:47:39.766 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:47:39.766 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:47:39.766 INFO:teuthology.orchestra.run.smithi121.stdout: mon_allow_pool_size_one = true 2024-03-29T11:47:39.766 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:47:39.766 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:47:39.766 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:47:39.766 INFO:teuthology.orchestra.run.smithi121.stdout: osd default data pool replay window = 5 2024-03-29T11:47:39.766 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:47:39.766 INFO:teuthology.orchestra.run.smithi121.stdout: mon allow pool delete = true 2024-03-29T11:47:39.766 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:47:39.767 INFO:teuthology.orchestra.run.smithi121.stdout: mon cluster log file level = debug 2024-03-29T11:47:39.767 INFO:teuthology.orchestra.run.smithi121.stdout: debug asserts on shutdown = true 2024-03-29T11:47:39.767 INFO:teuthology.orchestra.run.smithi121.stdout: mon health detail to clog = false 2024-03-29T11:47:39.767 INFO:teuthology.orchestra.run.smithi121.stdout: mon host = "[v1:172.21.15.98:6789],[v1:172.21.15.98:6790],[v1:172.21.15.121:6789]" 2024-03-29T11:47:39.767 INFO:teuthology.orchestra.run.smithi121.stdout: lockdep = True 2024-03-29T11:47:39.767 INFO:teuthology.orchestra.run.smithi121.stdout: mon client directed command retry = 5 2024-03-29T11:47:39.767 INFO:teuthology.orchestra.run.smithi121.stdout: mon election default strategy = 1 2024-03-29T11:47:39.767 INFO:teuthology.orchestra.run.smithi121.stdout: ms bind msgr2 = False 2024-03-29T11:47:39.767 INFO:teuthology.orchestra.run.smithi121.stdout: ms inject socket failures = 5000 2024-03-29T11:47:39.767 INFO:teuthology.orchestra.run.smithi121.stdout: ms type = async 2024-03-29T11:47:39.767 INFO:teuthology.orchestra.run.smithi121.stdout: fsid = 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:47:39.767 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:47:39.767 INFO:teuthology.orchestra.run.smithi121.stdout:[osd] 2024-03-29T11:47:39.767 INFO:teuthology.orchestra.run.smithi121.stdout: osd journal size = 100 2024-03-29T11:47:39.768 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:47:39.768 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub load threshold = 5.0 2024-03-29T11:47:39.768 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub max interval = 600 2024-03-29T11:47:39.768 INFO:teuthology.orchestra.run.smithi121.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:47:39.768 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:47:39.768 INFO:teuthology.orchestra.run.smithi121.stdout: osd recover clone overlap = true 2024-03-29T11:47:39.768 INFO:teuthology.orchestra.run.smithi121.stdout: osd recovery max chunk = 1048576 2024-03-29T11:47:39.768 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:47:39.768 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug shutdown = true 2024-03-29T11:47:39.768 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug op order = true 2024-03-29T11:47:39.768 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug verify stray on activate = true 2024-03-29T11:47:39.768 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug trim objects = true 2024-03-29T11:47:39.768 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:47:39.768 INFO:teuthology.orchestra.run.smithi121.stdout: osd open classes on start = true 2024-03-29T11:47:39.768 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug pg log writeout = true 2024-03-29T11:47:39.769 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:47:39.769 INFO:teuthology.orchestra.run.smithi121.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:47:39.769 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:47:39.769 INFO:teuthology.orchestra.run.smithi121.stdout: osd map max advance = 10 2024-03-29T11:47:39.769 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:47:39.769 INFO:teuthology.orchestra.run.smithi121.stdout: journal zero on create = true 2024-03-29T11:47:39.769 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:47:39.769 INFO:teuthology.orchestra.run.smithi121.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:47:39.769 INFO:teuthology.orchestra.run.smithi121.stdout: filestore apply finisher threads = 3 2024-03-29T11:47:39.769 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:47:39.769 INFO:teuthology.orchestra.run.smithi121.stdout: bdev debug aio = true 2024-03-29T11:47:39.769 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug misdirected ops = true 2024-03-29T11:47:39.769 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore block size = 96636764160 2024-03-29T11:47:39.769 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore compression algorithm = snappy 2024-03-29T11:47:39.770 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore compression mode = aggressive 2024-03-29T11:47:39.770 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore fsck on mount = True 2024-03-29T11:47:39.770 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore zero block detection = True 2024-03-29T11:47:39.770 INFO:teuthology.orchestra.run.smithi121.stdout: debug bluefs = 20 2024-03-29T11:47:39.770 INFO:teuthology.orchestra.run.smithi121.stdout: debug bluestore = 20 2024-03-29T11:47:39.770 INFO:teuthology.orchestra.run.smithi121.stdout: debug monc = 20 2024-03-29T11:47:39.770 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-03-29T11:47:39.770 INFO:teuthology.orchestra.run.smithi121.stdout: debug osd = 20 2024-03-29T11:47:39.770 INFO:teuthology.orchestra.run.smithi121.stdout: debug rocksdb = 10 2024-03-29T11:47:39.770 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:47:39.770 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd full ratio = 0.9 2024-03-29T11:47:39.770 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:47:39.770 INFO:teuthology.orchestra.run.smithi121.stdout: osd class default list = * 2024-03-29T11:47:39.770 INFO:teuthology.orchestra.run.smithi121.stdout: osd class load list = * 2024-03-29T11:47:39.770 INFO:teuthology.orchestra.run.smithi121.stdout: osd client watch timeout = 120 2024-03-29T11:47:39.771 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug verify cached snaps = True 2024-03-29T11:47:39.771 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug verify missing on start = True 2024-03-29T11:47:39.771 INFO:teuthology.orchestra.run.smithi121.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:47:39.771 INFO:teuthology.orchestra.run.smithi121.stdout: osd mclock override recovery settings = True 2024-03-29T11:47:39.771 INFO:teuthology.orchestra.run.smithi121.stdout: osd objectstore = bluestore 2024-03-29T11:47:39.771 INFO:teuthology.orchestra.run.smithi121.stdout: osd op queue = debug_random 2024-03-29T11:47:39.771 INFO:teuthology.orchestra.run.smithi121.stdout: osd op queue cut off = debug_random 2024-03-29T11:47:39.771 INFO:teuthology.orchestra.run.smithi121.stdout: osd shutdown pgref assert = True 2024-03-29T11:47:39.771 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:47:39.771 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr] 2024-03-29T11:47:39.771 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-03-29T11:47:39.771 INFO:teuthology.orchestra.run.smithi121.stdout: debug mgr = 20 2024-03-29T11:47:39.771 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2024-03-29T11:47:39.771 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2024-03-29T11:47:39.772 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:47:39.772 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:47:39.772 INFO:teuthology.orchestra.run.smithi121.stdout: mgr/telemetry/nag = false 2024-03-29T11:47:39.772 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:47:39.772 INFO:teuthology.orchestra.run.smithi121.stdout:[mon] 2024-03-29T11:47:39.772 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-03-29T11:47:39.772 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2024-03-29T11:47:39.772 INFO:teuthology.orchestra.run.smithi121.stdout: debug paxos = 20 2024-03-29T11:47:39.772 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2024-03-29T11:47:39.772 INFO:teuthology.orchestra.run.smithi121.stdout: mon data avail warn = 5 2024-03-29T11:47:39.772 INFO:teuthology.orchestra.run.smithi121.stdout: mon mgr mkfs grace = 240 2024-03-29T11:47:39.773 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:47:39.773 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd reporter subtree level = osd 2024-03-29T11:47:39.773 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd prime pg temp = true 2024-03-29T11:47:39.773 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:47:39.773 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:47:39.773 INFO:teuthology.orchestra.run.smithi121.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:47:39.773 INFO:teuthology.orchestra.run.smithi121.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:47:39.773 INFO:teuthology.orchestra.run.smithi121.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:47:39.773 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:47:39.773 INFO:teuthology.orchestra.run.smithi121.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:47:39.773 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:47:39.773 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:47:39.773 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:47:39.773 INFO:teuthology.orchestra.run.smithi121.stdout: # 1m isn't quite enough 2024-03-29T11:47:39.774 INFO:teuthology.orchestra.run.smithi121.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:47:39.774 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:47:39.774 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:47:39.774 INFO:teuthology.orchestra.run.smithi121.stdout: mon min osdmap epochs = 50 2024-03-29T11:47:39.774 INFO:teuthology.orchestra.run.smithi121.stdout: mon osdmap full prune interval = 2 2024-03-29T11:47:39.774 INFO:teuthology.orchestra.run.smithi121.stdout: mon osdmap full prune min = 15 2024-03-29T11:47:39.774 INFO:teuthology.orchestra.run.smithi121.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:47:39.774 INFO:teuthology.orchestra.run.smithi121.stdout: mon scrub interval = 300 2024-03-29T11:47:39.774 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on pool no app = False 2024-03-29T11:47:39.774 INFO:teuthology.orchestra.run.smithi121.stdout: paxos service trim min = 10 2024-03-29T11:47:39.774 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:47:39.774 INFO:teuthology.orchestra.run.smithi121.stdout:[client] 2024-03-29T11:47:39.774 INFO:teuthology.orchestra.run.smithi121.stdout: rgw cache enabled = true 2024-03-29T11:47:39.774 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable ops log = true 2024-03-29T11:47:39.774 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable usage log = true 2024-03-29T11:47:39.775 INFO:teuthology.orchestra.run.smithi121.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:47:39.775 INFO:teuthology.orchestra.run.smithi121.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:47:39.775 INFO:teuthology.orchestra.run.smithi121.stdout: debug monc = 20 2024-03-29T11:47:39.775 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-03-29T11:47:39.775 INFO:teuthology.orchestra.run.smithi121.stdout: debug objecter = 20 2024-03-29T11:47:39.775 INFO:teuthology.orchestra.run.smithi121.stdout: debug rados = 20 2024-03-29T11:47:39.775 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.a] 2024-03-29T11:47:39.775 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.c] 2024-03-29T11:47:39.775 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.b] 2024-03-29T11:47:39.777 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T11:47:39.777 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T11:47:39.881 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T11:47:39.881 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T11:47:39.881 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T11:47:39.900 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T11:47:39.900 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T11:47:39.960 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi098.front.sepia.ceph.com 2024-03-29T11:47:39.960 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T11:47:39.960 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:47:39.960 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:47:40.050 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T11:47:40.050 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:47:40.108 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi121.front.sepia.ceph.com 2024-03-29T11:47:40.109 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-29T11:47:40.109 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:47:40.109 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:47:40.148 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-29T11:47:40.149 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:47:40.207 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T11:47:40.207 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T11:47:40.208 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:47:40.265 INFO:teuthology.orchestra.run.smithi098.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:47:40.270 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:47:40.329 INFO:teuthology.orchestra.run.smithi121.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:47:40.334 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T11:47:40.335 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T11:47:40.335 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T11:47:40.378 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T11:47:40.398 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T11:47:40.441 INFO:teuthology.orchestra.run.smithi121.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T11:47:40.456 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T11:47:40.456 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi098.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi121.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:47:40.456 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T11:47:40.484 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:47:40.484 INFO:tasks.ceph:role: osd.0 2024-03-29T11:47:40.484 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi098.front.sepia.ceph.com 2024-03-29T11:47:40.484 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:47:40.560 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:47:40.560 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:47:40.560 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:47:40.560 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:47:40.560 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:47:40.561 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:47:40.561 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:47:40.561 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:47:40.561 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:47:40.561 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:47:41.026 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2024-03-29T11:47:41.028 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2024-03-29T11:47:41.029 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T11:47:41.148 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T11:47:41.184 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T11:47:41.251 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:47:41.251 INFO:tasks.ceph:role: osd.1 2024-03-29T11:47:41.251 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi098.front.sepia.ceph.com 2024-03-29T11:47:41.252 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:47:41.327 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:47:41.327 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:47:41.327 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:47:41.327 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:47:41.327 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:47:41.327 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:47:41.327 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:47:41.327 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:47:41.327 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:47:41.327 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:47:41.789 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2024-03-29T11:47:41.791 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2024-03-29T11:47:41.792 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T11:47:41.868 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T11:47:41.941 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T11:47:42.010 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:47:42.010 INFO:tasks.ceph:role: osd.2 2024-03-29T11:47:42.010 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi098.front.sepia.ceph.com 2024-03-29T11:47:42.010 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:47:42.084 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:47:42.084 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:47:42.084 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:47:42.084 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:47:42.084 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:47:42.084 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:47:42.084 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:47:42.085 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:47:42.085 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:47:42.085 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:47:42.577 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2024-03-29T11:47:42.579 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2024-03-29T11:47:42.579 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T11:47:42.678 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T11:47:42.711 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T11:47:42.778 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:47:42.779 INFO:tasks.ceph:role: osd.3 2024-03-29T11:47:42.779 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi098.front.sepia.ceph.com 2024-03-29T11:47:42.779 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:47:42.854 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:47:42.854 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:47:42.854 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:47:42.854 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:47:42.854 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:47:42.854 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:47:42.854 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:47:42.854 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:47:42.854 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:47:42.854 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:47:43.377 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2024-03-29T11:47:43.379 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2024-03-29T11:47:43.379 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T11:47:43.515 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T11:47:43.551 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:47:43.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:47:43.654+0000 7fd1fa981640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T11:47:43.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:47:43.656+0000 7fd1fa981640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T11:47:43.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:47:43.656+0000 7fd1fa981640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:47:43.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:47:43.656+0000 7fd1fa981640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:47:43.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:47:43.656+0000 7fd1fa981640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:47:43.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:47:43.661+0000 7fd1fa981640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T11:47:46.515 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T11:47:46.548 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:47:46.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:47:46.633+0000 7f1d09213640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T11:47:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:47:46.634+0000 7f1d09213640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T11:47:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:47:46.634+0000 7f1d09213640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:47:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:47:46.634+0000 7f1d09213640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:47:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:47:46.634+0000 7f1d09213640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:47:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:47:46.634+0000 7f1d09213640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T11:47:49.489 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T11:47:49.525 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:47:49.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:47:49.613+0000 7f595b897640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T11:47:49.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:47:49.614+0000 7f595b897640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T11:47:49.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:47:49.614+0000 7f595b897640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:47:49.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:47:49.614+0000 7f595b897640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:47:49.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:47:49.614+0000 7f595b897640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:47:49.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:47:49.614+0000 7f595b897640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T11:47:52.467 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T11:47:52.497 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:47:52.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:47:52.580+0000 7f0921811640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T11:47:52.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:47:52.581+0000 7f0921811640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T11:47:52.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:47:52.581+0000 7f0921811640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:47:52.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:47:52.581+0000 7f0921811640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:47:52.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:47:52.581+0000 7f0921811640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:47:52.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:47:52.581+0000 7f0921811640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T11:47:55.438 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T11:47:55.469 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T11:47:55.500 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:47:55.500 INFO:tasks.ceph:role: osd.4 2024-03-29T11:47:55.500 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi121.front.sepia.ceph.com 2024-03-29T11:47:55.500 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:47:55.578 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:47:55.578 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:47:55.579 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:47:55.579 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:47:55.579 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:47:55.579 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:47:55.579 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:47:55.579 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:47:55.580 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:47:55.580 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:47:56.059 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2024-03-29T11:47:56.062 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2024-03-29T11:47:56.062 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T11:47:56.205 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T11:47:56.242 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T11:47:56.312 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:47:56.312 INFO:tasks.ceph:role: osd.5 2024-03-29T11:47:56.312 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi121.front.sepia.ceph.com 2024-03-29T11:47:56.312 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:47:56.388 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:47:56.388 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:47:56.388 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:47:56.388 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:47:56.388 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:47:56.388 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:47:56.389 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:47:56.389 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:47:56.389 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:47:56.389 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:47:56.842 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2024-03-29T11:47:56.844 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2024-03-29T11:47:56.844 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T11:47:56.926 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T11:47:56.997 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T11:47:57.064 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:47:57.064 INFO:tasks.ceph:role: osd.6 2024-03-29T11:47:57.064 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi121.front.sepia.ceph.com 2024-03-29T11:47:57.064 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:47:57.138 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:47:57.139 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:47:57.139 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:47:57.139 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:47:57.139 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:47:57.139 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:47:57.139 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:47:57.139 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:47:57.139 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:47:57.139 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:47:57.649 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2024-03-29T11:47:57.651 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2024-03-29T11:47:57.651 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T11:47:57.781 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T11:47:57.811 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T11:47:57.878 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:47:57.878 INFO:tasks.ceph:role: osd.7 2024-03-29T11:47:57.878 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi121.front.sepia.ceph.com 2024-03-29T11:47:57.879 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:47:57.951 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:47:57.952 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:47:57.952 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:47:57.952 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:47:57.952 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:47:57.952 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:47:57.952 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:47:57.952 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:47:57.952 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:47:57.952 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:47:58.466 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2024-03-29T11:47:58.468 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2024-03-29T11:47:58.468 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T11:47:58.601 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T11:47:58.637 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:47:58.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:47:58.738+0000 7fe06552a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T11:47:58.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:47:58.738+0000 7fe06552a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T11:47:58.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:47:58.738+0000 7fe06552a640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:47:58.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:47:58.738+0000 7fe06552a640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:47:58.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:47:58.738+0000 7fe06552a640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:47:58.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:47:58.739+0000 7fe06552a640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T11:48:01.597 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T11:48:01.628 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:48:01.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:01.713+0000 7fbff3a13640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T11:48:01.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:01.713+0000 7fbff3a13640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T11:48:01.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:01.713+0000 7fbff3a13640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:48:01.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:01.713+0000 7fbff3a13640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:48:01.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:01.714+0000 7fbff3a13640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:48:01.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:01.714+0000 7fbff3a13640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T11:48:04.573 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T11:48:04.604 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:48:04.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:04.689+0000 7f6ff3404640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T11:48:04.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:04.689+0000 7f6ff3404640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T11:48:04.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:04.689+0000 7f6ff3404640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:48:04.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:04.689+0000 7f6ff3404640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:48:04.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:04.689+0000 7f6ff3404640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:48:04.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:04.690+0000 7f6ff3404640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T11:48:07.554 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T11:48:07.585 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:48:07.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:07.669+0000 7f2a3f824640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T11:48:07.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:07.669+0000 7f2a3f824640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T11:48:07.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:07.669+0000 7f2a3f824640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:48:07.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:07.669+0000 7f2a3f824640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:48:07.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:07.669+0000 7f2a3f824640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:48:07.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:07.670+0000 7f2a3f824640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T11:48:10.522 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T11:48:10.554 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T11:48:10.554 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T11:48:10.554 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T11:48:10.586 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T11:48:10.586 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T11:48:10.657 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T11:48:10.657 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T11:48:10.727 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T11:48:10.727 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T11:48:10.796 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T11:48:10.796 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T11:48:10.865 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-29T11:48:10.865 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T11:48:10.896 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-29T11:48:10.897 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T11:48:10.967 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-29T11:48:10.967 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T11:48:11.036 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-29T11:48:11.037 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T11:48:11.105 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-29T11:48:11.105 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T11:48:11.173 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T11:48:11.173 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T11:48:11.190 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-29T11:48:11.190 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T11:48:11.229 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T11:48:11.229 DEBUG:teuthology.orchestra.run.smithi098:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:48:11.232 DEBUG:teuthology.orchestra.run.smithi121:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:48:11.273 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr.y] 2024-03-29T11:48:11.273 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBcqgZmgVeUDxAAsQIm/VQ6oamChHTuiCo21g== 2024-03-29T11:48:11.273 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.0] 2024-03-29T11:48:11.273 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBfqgZmua0MJxAAR/vqosX2TJqprzt6653ZTg== 2024-03-29T11:48:11.273 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.1] 2024-03-29T11:48:11.273 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBiqgZmwFvUJRAA6wA9CXCm2/rHxzeaju4J/A== 2024-03-29T11:48:11.273 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.2] 2024-03-29T11:48:11.273 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBlqgZmG7qiJBAAZdD7ooMCODENvtcGDVna8w== 2024-03-29T11:48:11.273 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.3] 2024-03-29T11:48:11.273 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBoqgZmz0WrIhAAq3DfS1LqreQgSq4YQ35pzA== 2024-03-29T11:48:11.274 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr.x] 2024-03-29T11:48:11.274 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBcqgZmHxZjExAASJOfoTqkXXRjLu3IEerfyw== 2024-03-29T11:48:11.274 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.4] 2024-03-29T11:48:11.274 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBuqgZmaHkVLBAAefw2tSY94/Um5w8kGh6azw== 2024-03-29T11:48:11.274 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.5] 2024-03-29T11:48:11.274 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBxqgZmE6WZKhAAoVQo18VwtS9nY7AAuUKOvQ== 2024-03-29T11:48:11.274 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.6] 2024-03-29T11:48:11.274 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQB0qgZmCOQoKRAAw+qstbt25cW0jVAGPIHuFg== 2024-03-29T11:48:11.274 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.7] 2024-03-29T11:48:11.274 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQB3qgZm0QP0JxAA5vJjc20buZ7zVhvP4IyLzg== 2024-03-29T11:48:11.274 INFO:teuthology.orchestra.run.smithi098.stdout:[client.0] 2024-03-29T11:48:11.274 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBcqgZmpGRJFhAAOIeV0S0Qkn9Pkt4sw5venA== 2024-03-29T11:48:11.274 INFO:teuthology.orchestra.run.smithi098.stdout:[client.1] 2024-03-29T11:48:11.274 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBcqgZmZTEJGhAA+2R8Uv2ouZTplga0s2qyTw== 2024-03-29T11:48:11.296 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr.y] 2024-03-29T11:48:11.296 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQBcqgZmgVeUDxAAsQIm/VQ6oamChHTuiCo21g== 2024-03-29T11:48:11.296 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.0] 2024-03-29T11:48:11.296 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQBfqgZmua0MJxAAR/vqosX2TJqprzt6653ZTg== 2024-03-29T11:48:11.296 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.1] 2024-03-29T11:48:11.296 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQBiqgZmwFvUJRAA6wA9CXCm2/rHxzeaju4J/A== 2024-03-29T11:48:11.296 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.2] 2024-03-29T11:48:11.296 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQBlqgZmG7qiJBAAZdD7ooMCODENvtcGDVna8w== 2024-03-29T11:48:11.296 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.3] 2024-03-29T11:48:11.297 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQBoqgZmz0WrIhAAq3DfS1LqreQgSq4YQ35pzA== 2024-03-29T11:48:11.297 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr.x] 2024-03-29T11:48:11.297 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQBcqgZmHxZjExAASJOfoTqkXXRjLu3IEerfyw== 2024-03-29T11:48:11.297 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.4] 2024-03-29T11:48:11.297 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQBuqgZmaHkVLBAAefw2tSY94/Um5w8kGh6azw== 2024-03-29T11:48:11.297 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.5] 2024-03-29T11:48:11.297 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQBxqgZmE6WZKhAAoVQo18VwtS9nY7AAuUKOvQ== 2024-03-29T11:48:11.297 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.6] 2024-03-29T11:48:11.297 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQB0qgZmCOQoKRAAw+qstbt25cW0jVAGPIHuFg== 2024-03-29T11:48:11.297 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.7] 2024-03-29T11:48:11.297 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQB3qgZm0QP0JxAA5vJjc20buZ7zVhvP4IyLzg== 2024-03-29T11:48:11.297 INFO:teuthology.orchestra.run.smithi121.stdout:[client.0] 2024-03-29T11:48:11.297 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQBcqgZmpGRJFhAAOIeV0S0Qkn9Pkt4sw5venA== 2024-03-29T11:48:11.297 INFO:teuthology.orchestra.run.smithi121.stdout:[client.1] 2024-03-29T11:48:11.298 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQBcqgZmZTEJGhAA+2R8Uv2ouZTplga0s2qyTw== 2024-03-29T11:48:11.298 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:48:11.317 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:48:11.392 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:48:11.394 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:48:11.446 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:48:11.449 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:48:11.500 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:48:11.502 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:48:11.554 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:48:11.556 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:48:11.609 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:48:11.611 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:48:11.663 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:48:11.665 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:48:11.716 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:48:11.719 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:48:11.770 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:48:11.772 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:48:11.824 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:48:11.826 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:48:11.877 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:48:11.880 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:48:11.934 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:48:11.937 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:48:11.991 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T11:48:11.991 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T11:48:12.018 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:48:12.115 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T11:48:12.141 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T11:48:12.208 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:48:12.305 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T11:48:12.332 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T11:48:12.359 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:48:12.456 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T11:48:12.483 DEBUG:teuthology.orchestra.run.smithi098:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:48:12.485 DEBUG:teuthology.orchestra.run.smithi121:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:48:12.538 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T11:48:12.538 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T11:48:12.538 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T11:48:12.540 INFO:tasks.ceph.mon.a:Started 2024-03-29T11:48:12.540 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T11:48:12.541 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T11:48:12.542 INFO:tasks.ceph.mon.c:Started 2024-03-29T11:48:12.542 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T11:48:12.543 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T11:48:12.580 INFO:tasks.ceph.mon.b:Started 2024-03-29T11:48:12.580 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T11:48:12.581 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T11:48:12.581 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T11:48:12.583 INFO:tasks.ceph.mgr.y:Started 2024-03-29T11:48:12.583 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T11:48:12.583 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T11:48:12.585 INFO:tasks.ceph.mgr.x:Started 2024-03-29T11:48:12.585 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T11:48:12.586 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T11:48:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:12.684+0000 7fc0430d3640 10 monclient: get_monmap_and_config 2024-03-29T11:48:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:12.684+0000 7fc0430d3640 10 monclient: build_initial_monmap 2024-03-29T11:48:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:12.684+0000 7fc0430d3640 10 monclient: monmap: 2024-03-29T11:48:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-03-29T11:48:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-29T11:48:12.685837+0000 2024-03-29T11:48:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-29T11:48:12.685837+0000 2024-03-29T11:48:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:48:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:48:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:48:12.691 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-29T11:48:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:12.685+0000 7fc0430d3640 1 Processor -- start 2024-03-29T11:48:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:12.685+0000 7fc0430d3640 1 -- start start 2024-03-29T11:48:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:12.685+0000 7fc0430d3640 10 monclient: init 2024-03-29T11:48:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:12.686+0000 7fc0430d3640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:12.686+0000 7fc0430d3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:48:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:12.686+0000 7fc0430d3640 10 monclient(hunting): picked mon.noname-c con 0x7fc03c0b8240 addr v1:172.21.15.121:6789/0 2024-03-29T11:48:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:12.686+0000 7fc0430d3640 10 monclient(hunting): picked mon.noname-a con 0x7fc03c0b1f60 addr v1:172.21.15.98:6789/0 2024-03-29T11:48:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:12.686+0000 7fc0430d3640 10 monclient(hunting): picked mon.noname-b con 0x7fc03c09c8c0 addr v1:172.21.15.98:6790/0 2024-03-29T11:48:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:12.686+0000 7fc0430d3640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc03c09bd80 con 0x7fc03c0b1f60 2024-03-29T11:48:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:12.686+0000 7fc0430d3640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc03c09b960 con 0x7fc03c0b8240 2024-03-29T11:48:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:12.686+0000 7fc0430d3640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc03c0b2ec0 con 0x7fc03c09c8c0 2024-03-29T11:48:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:12.686+0000 7fc0430d3640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:12.686+0000 7fc0430d3640 10 monclient(hunting): authenticate will time out at 799.771557s 2024-03-29T11:48:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:12.686+0000 7fc041670640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fc03c09c8c0 0x7fc03c14d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:41922/0 (socket says 172.21.15.98:41922) 2024-03-29T11:48:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:12.686+0000 7fc041670640 1 -- 172.21.15.98:0/1873034761 learned_addr learned my addr 172.21.15.98:0/1873034761 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:48:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:12.686+0000 7fc03bfff640 1 --1- 172.21.15.98:0/1873034761 >> v1:172.21.15.98:6789/0 conn(0x7fc03c0b1f60 0x7fc03c09c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43532/0 (socket says 172.21.15.98:43532) 2024-03-29T11:48:15.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:15.686+0000 7fc03a7fc640 10 monclient(hunting): tick 2024-03-29T11:48:15.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:15.686+0000 7fc03a7fc640 10 monclient(hunting): _check_auth_tickets 2024-03-29T11:48:15.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:15.686+0000 7fc03a7fc640 1 monclient(hunting): continuing hunt 2024-03-29T11:48:15.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:15.686+0000 7fc03a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T11:48:15.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:15.686+0000 7fc03a7fc640 1 -- 172.21.15.98:0/1873034761 >> v1:172.21.15.98:6790/0 conn(0x7fc03c09c8c0 legacy=0x7fc03c14d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:15.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:15.686+0000 7fc03a7fc640 1 -- 172.21.15.98:0/1873034761 >> v1:172.21.15.121:6789/0 conn(0x7fc03c0b8240 legacy=0x7fc03c13f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:15.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:15.686+0000 7fc03a7fc640 1 -- 172.21.15.98:0/1873034761 >> v1:172.21.15.98:6789/0 conn(0x7fc03c0b1f60 legacy=0x7fc03c09c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:15.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:15.686+0000 7fc03a7fc640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:48:15.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:15.686+0000 7fc03a7fc640 10 monclient(hunting): picked mon.noname-b con 0x7fc0280036e0 addr v1:172.21.15.98:6790/0 2024-03-29T11:48:15.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:15.686+0000 7fc03a7fc640 10 monclient(hunting): picked mon.noname-a con 0x7fc0280072d0 addr v1:172.21.15.98:6789/0 2024-03-29T11:48:15.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:15.686+0000 7fc03a7fc640 10 monclient(hunting): picked mon.noname-c con 0x7fc02800b000 addr v1:172.21.15.121:6789/0 2024-03-29T11:48:15.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:15.686+0000 7fc03a7fc640 1 -- 172.21.15.98:0/1873034761 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc03c13fb60 con 0x7fc0280072d0 2024-03-29T11:48:15.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:15.686+0000 7fc03a7fc640 1 -- 172.21.15.98:0/1873034761 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc02800fe10 con 0x7fc02800b000 2024-03-29T11:48:15.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:15.686+0000 7fc03a7fc640 1 -- 172.21.15.98:0/1873034761 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc028011050 con 0x7fc0280036e0 2024-03-29T11:48:15.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:15.686+0000 7fc03a7fc640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:17.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.857+0000 7fc03b7fe640 1 -- 172.21.15.98:0/1873034761 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3705465015 0 0) 0x7fc03c13fb60 con 0x7fc0280072d0 2024-03-29T11:48:17.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.857+0000 7fc03b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:48:17.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.857+0000 7fc03b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:17.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.857+0000 7fc03b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:17.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.857+0000 7fc03b7fe640 10 monclient(hunting): my global_id is 4101 2024-03-29T11:48:17.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.857+0000 7fc03b7fe640 1 -- 172.21.15.98:0/1873034761 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc024003520 con 0x7fc0280072d0 2024-03-29T11:48:17.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.858+0000 7fc03b7fe640 1 -- 172.21.15.98:0/1873034761 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1365475108 0 0) 0x7fc02800fe10 con 0x7fc02800b000 2024-03-29T11:48:17.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.858+0000 7fc03b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:48:17.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.858+0000 7fc03b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:17.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.858+0000 7fc03b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:17.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.858+0000 7fc03b7fe640 10 monclient(hunting): my global_id is 4105 2024-03-29T11:48:17.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.858+0000 7fc03b7fe640 1 -- 172.21.15.98:0/1873034761 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc03c13fb60 con 0x7fc02800b000 2024-03-29T11:48:17.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.858+0000 7fc03b7fe640 1 -- 172.21.15.98:0/1873034761 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2761078617 0 0) 0x7fc03c13fb60 con 0x7fc02800b000 2024-03-29T11:48:17.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.858+0000 7fc03b7fe640 1 -- 172.21.15.98:0/1873034761 >> v1:172.21.15.98:6790/0 conn(0x7fc0280036e0 legacy=0x7fc028005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:17.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.858+0000 7fc03b7fe640 1 -- 172.21.15.98:0/1873034761 >> v1:172.21.15.98:6789/0 conn(0x7fc0280072d0 legacy=0x7fc028009760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:17.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.858+0000 7fc03b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:48:17.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.858+0000 7fc03b7fe640 1 monclient: found mon.noname-c 2024-03-29T11:48:17.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.858+0000 7fc03b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:17.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.858+0000 7fc03b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:48:17.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.858+0000 7fc03b7fe640 1 -- 172.21.15.98:0/1873034761 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc028012290 con 0x7fc02800b000 2024-03-29T11:48:17.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.858+0000 7fc03b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:48:17.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.858+0000 7fc03b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:48:17.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.858+0000 7fc03b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:17.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.858+0000 7fc03b7fe640 1 -- 172.21.15.98:0/1873034761 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fc034001990 con 0x7fc02800b000 2024-03-29T11:48:17.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.858+0000 7fc03b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:17.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.858+0000 7fc03b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:48:17.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.858+0000 7fc03b7fe640 10 monclient: dump: 2024-03-29T11:48:17.868 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-29T11:48:17.868 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:17.868 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:17.868 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:17.868 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:17.868 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:17.868 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:17.869 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:17.869 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:17.869 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-29T11:48:17.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.858+0000 7fc0430d3640 5 monclient: authenticate success, global_id 4105 2024-03-29T11:48:17.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.858+0000 7fc0430d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:48:17.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.858+0000 7fc03b7fe640 1 -- 172.21.15.98:0/1873034761 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc034001af0 con 0x7fc02800b000 2024-03-29T11:48:17.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.858+0000 7fc03b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:17.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.858+0000 7fc03b7fe640 1 -- 172.21.15.98:0/1873034761 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fc034001f00 con 0x7fc02800b000 2024-03-29T11:48:17.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.858+0000 7fc03b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:17.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.858+0000 7fc03b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:48:17.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.858+0000 7fc03b7fe640 10 monclient: dump: 2024-03-29T11:48:17.869 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-29T11:48:17.869 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:17.869 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:17.870 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:17.870 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:17.870 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:17.870 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:17.870 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:17.870 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:17.870 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-29T11:48:17.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.858+0000 7fc0430d3640 10 monclient: get_monmap_and_config success 2024-03-29T11:48:17.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.858+0000 7fc0430d3640 4 set_mon_vals no callback set 2024-03-29T11:48:17.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.859+0000 7fc0430d3640 10 monclient: shutdown 2024-03-29T11:48:17.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.859+0000 7fc0430d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:17.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.859+0000 7fc0430d3640 1 -- 172.21.15.98:0/1873034761 >> v1:172.21.15.121:6789/0 conn(0x7fc02800b000 legacy=0x7fc02800d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:17.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.859+0000 7fc0430d3640 1 -- 172.21.15.98:0/1873034761 shutdown_connections 2024-03-29T11:48:17.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.859+0000 7fc0430d3640 1 -- 172.21.15.98:0/1873034761 >> 172.21.15.98:0/1873034761 conn(0x7fc03c066b00 msgr2=0x7fc03c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:17.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.859+0000 7fc0430d3640 1 -- 172.21.15.98:0/1873034761 shutdown_connections 2024-03-29T11:48:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.859+0000 7fc0430d3640 1 -- 172.21.15.98:0/1873034761 wait complete. 2024-03-29T11:48:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.860+0000 7fc0430d3640 10 monclient: build_initial_monmap 2024-03-29T11:48:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.860+0000 7fc0430d3640 10 monclient: monmap: 2024-03-29T11:48:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-03-29T11:48:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:48:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-03-29T11:48:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-03-29T11:48:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:48:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:48:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:48:17.871 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-29T11:48:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.860+0000 7fc0430d3640 1 librados: starting msgr at 2024-03-29T11:48:17.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.861+0000 7fc0430d3640 1 librados: starting objecter 2024-03-29T11:48:17.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.861+0000 7fc0430d3640 1 Processor -- start 2024-03-29T11:48:17.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.861+0000 7fc0430d3640 1 -- start start 2024-03-29T11:48:17.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.861+0000 7fc0430d3640 1 librados: setting wanted keys 2024-03-29T11:48:17.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.861+0000 7fc0430d3640 1 librados: calling monclient init 2024-03-29T11:48:17.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.861+0000 7fc0430d3640 10 monclient: init 2024-03-29T11:48:17.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.862+0000 7fc0430d3640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:17.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.862+0000 7fc0430d3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:48:17.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.862+0000 7fc0430d3640 10 monclient(hunting): picked mon.noname-b con 0x7fc0280036e0 addr v1:172.21.15.121:6789/0 2024-03-29T11:48:17.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.862+0000 7fc0430d3640 10 monclient(hunting): picked mon.noname-c con 0x7fc0280072d0 addr v1:172.21.15.98:6790/0 2024-03-29T11:48:17.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.862+0000 7fc0430d3640 10 monclient(hunting): picked mon.noname-a con 0x7fc02800b000 addr v1:172.21.15.98:6789/0 2024-03-29T11:48:17.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.862+0000 7fc0430d3640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc03c0afff0 con 0x7fc02800b000 2024-03-29T11:48:17.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.862+0000 7fc0430d3640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc03c0acfa0 con 0x7fc0280036e0 2024-03-29T11:48:17.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.862+0000 7fc0430d3640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc03c0ad130 con 0x7fc0280072d0 2024-03-29T11:48:17.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.862+0000 7fc0430d3640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:17.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.862+0000 7fc0430d3640 10 monclient(hunting): authenticate will time out at 804.947529s 2024-03-29T11:48:17.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.862+0000 7fc040e6f640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fc0280036e0 0x7fc03c0b0480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:40852/0 (socket says 172.21.15.98:40852) 2024-03-29T11:48:17.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.862+0000 7fc03bfff640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fc0280072d0 0x7fc03c139110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:49568/0 (socket says 172.21.15.98:49568) 2024-03-29T11:48:17.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.862+0000 7fc040e6f640 1 -- 172.21.15.98:0/1281255160 learned_addr learned my addr 172.21.15.98:0/1281255160 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:48:17.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.862+0000 7fc0397fa640 10 client.?.objecter ms_handle_connect 0x7fc0280036e0 2024-03-29T11:48:17.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.862+0000 7fc0397fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:17.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.862+0000 7fc0397fa640 10 client.?.objecter ms_handle_connect 0x7fc0280072d0 2024-03-29T11:48:17.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.862+0000 7fc0397fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:17.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.862+0000 7fc0397fa640 10 client.?.objecter ms_handle_connect 0x7fc02800b000 2024-03-29T11:48:17.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.862+0000 7fc0397fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:17.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.862+0000 7fc0397fa640 1 -- 172.21.15.98:0/1281255160 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2381969305 0 0) 0x7fc03c0acfa0 con 0x7fc0280036e0 2024-03-29T11:48:17.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.863+0000 7fc0397fa640 10 monclient(hunting): _negotiate 2024-03-29T11:48:17.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.863+0000 7fc0397fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:17.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.863+0000 7fc0397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:17.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.863+0000 7fc0397fa640 10 monclient(hunting): my global_id is 4114 2024-03-29T11:48:17.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.863+0000 7fc0397fa640 1 -- 172.21.15.98:0/1281255160 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0280119a0 con 0x7fc0280036e0 2024-03-29T11:48:17.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.863+0000 7fc0397fa640 1 -- 172.21.15.98:0/1281255160 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2948960867 0 0) 0x7fc03c0ad130 con 0x7fc0280072d0 2024-03-29T11:48:17.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.863+0000 7fc0397fa640 10 monclient(hunting): _negotiate 2024-03-29T11:48:17.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.863+0000 7fc0397fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:17.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.863+0000 7fc0397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:17.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.863+0000 7fc0397fa640 10 monclient(hunting): my global_id is 4106 2024-03-29T11:48:17.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.863+0000 7fc0397fa640 1 -- 172.21.15.98:0/1281255160 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc03c0acfa0 con 0x7fc0280072d0 2024-03-29T11:48:17.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.863+0000 7fc0397fa640 1 -- 172.21.15.98:0/1281255160 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2929585472 0 0) 0x7fc03c0afff0 con 0x7fc02800b000 2024-03-29T11:48:17.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.863+0000 7fc0397fa640 10 monclient(hunting): _negotiate 2024-03-29T11:48:17.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.863+0000 7fc0397fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:17.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.863+0000 7fc0397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:17.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.863+0000 7fc0397fa640 10 monclient(hunting): my global_id is 4107 2024-03-29T11:48:17.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.863+0000 7fc0397fa640 1 -- 172.21.15.98:0/1281255160 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc03c0ad130 con 0x7fc02800b000 2024-03-29T11:48:17.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.863+0000 7fc0397fa640 1 -- 172.21.15.98:0/1281255160 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1284108685 0 0) 0x7fc03c0ad130 con 0x7fc02800b000 2024-03-29T11:48:17.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.863+0000 7fc0397fa640 1 -- 172.21.15.98:0/1281255160 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc03c0afff0 con 0x7fc02800b000 2024-03-29T11:48:17.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.863+0000 7fc0397fa640 1 -- 172.21.15.98:0/1281255160 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fc034001e00 con 0x7fc02800b000 2024-03-29T11:48:17.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.863+0000 7fc0397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:17.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.863+0000 7fc0397fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:48:17.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.863+0000 7fc0397fa640 10 monclient(hunting): dump: 2024-03-29T11:48:17.880 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-29T11:48:17.880 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:17.880 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:17.880 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:17.880 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:17.880 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:17.880 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:17.880 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:17.880 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:17.880 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-29T11:48:17.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.863+0000 7fc0397fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:48:17.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.864+0000 7fc0397fa640 1 -- 172.21.15.98:0/1281255160 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2694239576 0 0) 0x7fc03c0afff0 con 0x7fc02800b000 2024-03-29T11:48:17.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.864+0000 7fc0397fa640 1 -- 172.21.15.98:0/1281255160 >> v1:172.21.15.98:6790/0 conn(0x7fc0280072d0 legacy=0x7fc03c139110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:17.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.864+0000 7fc0397fa640 1 -- 172.21.15.98:0/1281255160 >> v1:172.21.15.121:6789/0 conn(0x7fc0280036e0 legacy=0x7fc03c0b0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:17.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.864+0000 7fc0397fa640 10 monclient: _finish_hunting 0 2024-03-29T11:48:17.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.864+0000 7fc0397fa640 1 monclient: found mon.a 2024-03-29T11:48:17.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.864+0000 7fc0397fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:17.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.864+0000 7fc0397fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:48:17.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.864+0000 7fc0397fa640 1 -- 172.21.15.98:0/1281255160 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc03c1db3a0 con 0x7fc02800b000 2024-03-29T11:48:17.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.864+0000 7fc0397fa640 10 monclient: _finish_auth 0 2024-03-29T11:48:17.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.864+0000 7fc0397fa640 10 monclient: _check_auth_tickets 2024-03-29T11:48:17.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.864+0000 7fc0397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:17.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.864+0000 7fc0430d3640 5 monclient: authenticate success, global_id 4107 2024-03-29T11:48:17.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.864+0000 7fc0430d3640 10 monclient: _renew_subs 2024-03-29T11:48:17.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.864+0000 7fc0430d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:48:17.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.864+0000 7fc0430d3640 1 -- 172.21.15.98:0/1281255160 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc03c1db590 con 0x7fc02800b000 2024-03-29T11:48:17.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.864+0000 7fc0430d3640 10 client.4107.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:48:17.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.864+0000 7fc0430d3640 10 monclient: _renew_subs 2024-03-29T11:48:17.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.864+0000 7fc0430d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:48:17.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.864+0000 7fc0430d3640 1 -- 172.21.15.98:0/1281255160 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc03c0b0d70 con 0x7fc02800b000 2024-03-29T11:48:17.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.864+0000 7fc0430d3640 1 librados: init done 2024-03-29T11:48:17.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.865+0000 7fc0430d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:17.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.865+0000 7fc0430d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:17.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.865+0000 7fc0430d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:48:17.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.865+0000 7fc0430d3640 1 -- 172.21.15.98:0/1281255160 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc03c09b170 con 0x7fc02800b000 2024-03-29T11:48:17.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.866+0000 7fc0397fa640 1 -- 172.21.15.98:0/1281255160 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc034001510 con 0x7fc02800b000 2024-03-29T11:48:17.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.866+0000 7fc0397fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:17.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.866+0000 7fc0397fa640 1 -- 172.21.15.98:0/1281255160 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fc034004d00 con 0x7fc02800b000 2024-03-29T11:48:17.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.866+0000 7fc0397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:17.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.866+0000 7fc0397fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:48:17.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.866+0000 7fc0397fa640 10 monclient: dump: 2024-03-29T11:48:17.883 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-29T11:48:17.883 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:17.883 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:17.883 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:17.883 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:17.883 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:17.883 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:17.883 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:17.883 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:17.883 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-29T11:48:17.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.866+0000 7fc0397fa640 1 -- 172.21.15.98:0/1281255160 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc0340052b0 con 0x7fc02800b000 2024-03-29T11:48:17.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.866+0000 7fc0397fa640 1 -- 172.21.15.98:0/1281255160 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 1528998711 0 0) 0x7fc034006750 con 0x7fc02800b000 2024-03-29T11:48:17.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.866+0000 7fc0397fa640 10 client.4107.objecter ms_dispatch 0x7fc03c000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T11:48:17.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.866+0000 7fc0397fa640 3 client.4107.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T11:48:17.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.866+0000 7fc0397fa640 3 client.4107.objecter handle_osd_map decoding full epoch 1 2024-03-29T11:48:17.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.866+0000 7fc0397fa640 20 client.4107.objecter dump_active .. 0 homeless 2024-03-29T11:48:17.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.866+0000 7fc03affd640 4 set_mon_vals no callback set 2024-03-29T11:48:17.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.868+0000 7fc0397fa640 1 -- 172.21.15.98:0/1281255160 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fc034006ad0 con 0x7fc02800b000 2024-03-29T11:48:17.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.868+0000 7fc0397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:17.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.868+0000 7fc0397fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:48:17.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.941+0000 7fc0430d3640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:48:17.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.941+0000 7fc0430d3640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:48:17.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.941+0000 7fc0430d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:48:17.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.941+0000 7fc0430d3640 1 -- 172.21.15.98:0/1281255160 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fc03c04f730 con 0x7fc02800b000 2024-03-29T11:48:17.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.945+0000 7fc0397fa640 1 -- 172.21.15.98:0/1281255160 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7fc034005640 con 0x7fc02800b000 2024-03-29T11:48:17.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.945+0000 7fc0397fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:48:17.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.945+0000 7fc0397fa640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T11:48:17.951 INFO:teuthology.orchestra.run.smithi098.stderr:adjusted tunables profile to default 2024-03-29T11:48:17.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.946+0000 7fc0430d3640 10 librados: watch_flush enter 2024-03-29T11:48:17.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.946+0000 7fc0430d3640 10 librados: watch_flush exit 2024-03-29T11:48:17.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.946+0000 7fc0430d3640 20 client.4107.objecter shutdown clearing up homeless session... 2024-03-29T11:48:17.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.946+0000 7fc0430d3640 10 client.4107.objecter successfully canceled tick 2024-03-29T11:48:17.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.947+0000 7fc0430d3640 10 monclient: shutdown 2024-03-29T11:48:17.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.947+0000 7fc0430d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:17.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.947+0000 7fc0430d3640 1 -- 172.21.15.98:0/1281255160 >> v1:172.21.15.98:6789/0 conn(0x7fc02800b000 legacy=0x7fc03c1e2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:17.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.947+0000 7fc0430d3640 1 -- 172.21.15.98:0/1281255160 shutdown_connections 2024-03-29T11:48:17.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.947+0000 7fc0430d3640 1 -- 172.21.15.98:0/1281255160 >> 172.21.15.98:0/1281255160 conn(0x7fc03c066b00 msgr2=0x7fc03c06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:17.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.947+0000 7fc0430d3640 1 -- 172.21.15.98:0/1281255160 shutdown_connections 2024-03-29T11:48:17.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.947+0000 7fc0430d3640 1 -- 172.21.15.98:0/1281255160 wait complete. 2024-03-29T11:48:17.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:17.947+0000 7fc0430d3640 1 librados: shutdown 2024-03-29T11:48:17.967 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T11:48:17.967 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T11:48:17.967 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T11:48:17.967 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T11:48:17.994 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T11:48:17.994 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T11:48:18.061 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T11:48:18.061 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T11:48:18.128 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T11:48:18.128 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T11:48:18.195 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-29T11:48:18.195 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T11:48:18.221 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-29T11:48:18.221 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T11:48:18.288 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-29T11:48:18.288 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T11:48:18.355 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-29T11:48:18.355 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T11:48:18.361 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T11:48:18.356+0000 7faaea042040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:48:18.396 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:48:18.391+0000 7f4e33c10040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:48:18.415 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T11:48:18.410+0000 7faaea042040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:48:18.422 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new c3764135-6315-4c21-b430-e3515422b4a5 0 2024-03-29T11:48:18.455 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:48:18.448+0000 7f4e33c10040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:48:18.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.548+0000 7f601a9e6640 10 monclient: get_monmap_and_config 2024-03-29T11:48:18.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.548+0000 7f601a9e6640 10 monclient: build_initial_monmap 2024-03-29T11:48:18.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.548+0000 7f601a9e6640 10 monclient: monmap: 2024-03-29T11:48:18.554 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-29T11:48:18.554 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:18.554 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:48:18.549442+0000 2024-03-29T11:48:18.554 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:48:18.549442+0000 2024-03-29T11:48:18.554 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:18.554 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:18.554 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:48:18.555 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:48:18.555 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:48:18.555 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:18.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.549+0000 7f601a9e6640 1 Processor -- start 2024-03-29T11:48:18.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.549+0000 7f601a9e6640 1 -- start start 2024-03-29T11:48:18.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.549+0000 7f601a9e6640 10 monclient: init 2024-03-29T11:48:18.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.549+0000 7f601a9e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:18.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.549+0000 7f601a9e6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:48:18.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.549+0000 7f601a9e6640 10 monclient(hunting): picked mon.noname-c con 0x7f601409ac40 addr v1:172.21.15.121:6789/0 2024-03-29T11:48:18.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.549+0000 7f601a9e6640 10 monclient(hunting): picked mon.noname-a con 0x7f601409c870 addr v1:172.21.15.98:6789/0 2024-03-29T11:48:18.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.549+0000 7f601a9e6640 10 monclient(hunting): picked mon.noname-b con 0x7f60140b4360 addr v1:172.21.15.98:6790/0 2024-03-29T11:48:18.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.549+0000 7f601a9e6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f601409dec0 con 0x7f601409c870 2024-03-29T11:48:18.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.549+0000 7f601a9e6640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f601409da30 con 0x7f601409ac40 2024-03-29T11:48:18.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.549+0000 7f601a9e6640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60140b39c0 con 0x7f60140b4360 2024-03-29T11:48:18.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.549+0000 7f601a9e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:18.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.549+0000 7f601a9e6640 10 monclient(hunting): authenticate will time out at 808.720023s 2024-03-29T11:48:18.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.549+0000 7f6013fff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f601409ac40 0x7f60140b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:51422/0 (socket says 172.21.15.121:51422) 2024-03-29T11:48:18.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.549+0000 7f6013fff640 1 -- 172.21.15.121:0/2152679724 learned_addr learned my addr 172.21.15.121:0/2152679724 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-29T11:48:18.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f6012ffd640 1 -- 172.21.15.121:0/2152679724 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3570284745 0 0) 0x7f601409da30 con 0x7f601409ac40 2024-03-29T11:48:18.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f6012ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:48:18.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f6012ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:18.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f6012ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:18.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f6012ffd640 10 monclient(hunting): my global_id is 4117 2024-03-29T11:48:18.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f6012ffd640 1 -- 172.21.15.121:0/2152679724 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ffc003520 con 0x7f601409ac40 2024-03-29T11:48:18.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f6012ffd640 1 -- 172.21.15.121:0/2152679724 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 853529522 0 0) 0x7f60140b39c0 con 0x7f60140b4360 2024-03-29T11:48:18.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f6012ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:48:18.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f6012ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:18.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f6012ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:18.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f6012ffd640 10 monclient(hunting): my global_id is 4109 2024-03-29T11:48:18.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f6012ffd640 1 -- 172.21.15.121:0/2152679724 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f601409da30 con 0x7f60140b4360 2024-03-29T11:48:18.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f6012ffd640 1 -- 172.21.15.121:0/2152679724 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2318746839 0 0) 0x7f6008001110 con 0x7f601409c870 2024-03-29T11:48:18.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f6012ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:48:18.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f6012ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:18.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f6012ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:18.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f6012ffd640 10 monclient(hunting): my global_id is 4110 2024-03-29T11:48:18.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f6012ffd640 1 -- 172.21.15.121:0/2152679724 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60140b39c0 con 0x7f601409c870 2024-03-29T11:48:18.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f6012ffd640 1 -- 172.21.15.121:0/2152679724 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 606628970 0 0) 0x7f5ffc003520 con 0x7f601409ac40 2024-03-29T11:48:18.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f6012ffd640 1 -- 172.21.15.121:0/2152679724 >> v1:172.21.15.98:6790/0 conn(0x7f60140b4360 legacy=0x7f601414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:18.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f6012ffd640 1 -- 172.21.15.121:0/2152679724 >> v1:172.21.15.98:6789/0 conn(0x7f601409c870 legacy=0x7f601413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:18.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f6012ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:48:18.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f6012ffd640 1 monclient: found mon.noname-c 2024-03-29T11:48:18.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f6012ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:18.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f6012ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:48:18.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f6012ffd640 1 -- 172.21.15.121:0/2152679724 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60140b5160 con 0x7f601409ac40 2024-03-29T11:48:18.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f6012ffd640 10 monclient: _finish_auth 0 2024-03-29T11:48:18.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f6012ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:48:18.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f6012ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:18.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f601a9e6640 5 monclient: authenticate success, global_id 4117 2024-03-29T11:48:18.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f6012ffd640 1 -- 172.21.15.121:0/2152679724 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f6004001690 con 0x7f601409ac40 2024-03-29T11:48:18.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f6012ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:18.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f6012ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:48:18.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f6012ffd640 10 monclient: dump: 2024-03-29T11:48:18.572 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:18.572 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:18.572 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:18.573 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:18.573 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:18.573 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:18.573 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:18.573 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:18.573 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:18.574 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:18.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f601a9e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:48:18.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f6012ffd640 1 -- 172.21.15.121:0/2152679724 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60040017f0 con 0x7f601409ac40 2024-03-29T11:48:18.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f6012ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:18.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f601a9e6640 10 monclient: get_monmap_and_config success 2024-03-29T11:48:18.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f6012ffd640 1 -- 172.21.15.121:0/2152679724 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f6004001c00 con 0x7f601409ac40 2024-03-29T11:48:18.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f6012ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:18.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f601a9e6640 4 set_mon_vals no callback set 2024-03-29T11:48:18.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f6012ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:48:18.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f6012ffd640 10 monclient: dump: 2024-03-29T11:48:18.575 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:18.575 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:18.576 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:18.576 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:18.576 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:18.576 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:18.576 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:18.576 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:18.577 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:18.577 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:18.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f601a9e6640 10 monclient: shutdown 2024-03-29T11:48:18.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f601a9e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:18.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f601a9e6640 1 -- 172.21.15.121:0/2152679724 >> v1:172.21.15.121:6789/0 conn(0x7f601409ac40 legacy=0x7f60140b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:18.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.550+0000 7f601a9e6640 1 -- 172.21.15.121:0/2152679724 shutdown_connections 2024-03-29T11:48:18.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.551+0000 7f601a9e6640 1 -- 172.21.15.121:0/2152679724 >> 172.21.15.121:0/2152679724 conn(0x7f6014066b00 msgr2=0x7f6014066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:18.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.551+0000 7f601a9e6640 1 -- 172.21.15.121:0/2152679724 shutdown_connections 2024-03-29T11:48:18.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.551+0000 7f601a9e6640 1 -- 172.21.15.121:0/2152679724 wait complete. 2024-03-29T11:48:18.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.551+0000 7f601a9e6640 10 monclient: build_initial_monmap 2024-03-29T11:48:18.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.551+0000 7f601a9e6640 10 monclient: monmap: 2024-03-29T11:48:18.579 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-29T11:48:18.579 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:48:18.579 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-03-29T11:48:18.579 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-03-29T11:48:18.579 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:18.580 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:18.580 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:48:18.580 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:48:18.580 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:48:18.580 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:18.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.551+0000 7f601a9e6640 1 librados: starting msgr at 2024-03-29T11:48:18.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.551+0000 7f601a9e6640 1 librados: starting objecter 2024-03-29T11:48:18.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.551+0000 7f601a9e6640 1 Processor -- start 2024-03-29T11:48:18.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.552+0000 7f601a9e6640 1 -- start start 2024-03-29T11:48:18.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.552+0000 7f601a9e6640 1 librados: setting wanted keys 2024-03-29T11:48:18.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.552+0000 7f601a9e6640 1 librados: calling monclient init 2024-03-29T11:48:18.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.552+0000 7f601a9e6640 10 monclient: init 2024-03-29T11:48:18.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.552+0000 7f601a9e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:18.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.552+0000 7f601a9e6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:48:18.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.552+0000 7f601a9e6640 10 monclient(hunting): picked mon.noname-b con 0x7f601409ac40 addr v1:172.21.15.121:6789/0 2024-03-29T11:48:18.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.552+0000 7f601a9e6640 10 monclient(hunting): picked mon.noname-c con 0x7f60140ad140 addr v1:172.21.15.98:6790/0 2024-03-29T11:48:18.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.552+0000 7f601a9e6640 10 monclient(hunting): picked mon.noname-a con 0x7f60141d77e0 addr v1:172.21.15.98:6789/0 2024-03-29T11:48:18.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.552+0000 7f601a9e6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60141d7c20 con 0x7f60141d77e0 2024-03-29T11:48:18.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.552+0000 7f601a9e6640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60141d7db0 con 0x7f601409ac40 2024-03-29T11:48:18.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.552+0000 7f601a9e6640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60141d7f40 con 0x7f60140ad140 2024-03-29T11:48:18.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.552+0000 7f601a9e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:18.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.552+0000 7f601a9e6640 10 monclient(hunting): authenticate will time out at 808.722930s 2024-03-29T11:48:18.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.552+0000 7f6013fff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f601409ac40 0x7f60140b09f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:51426/0 (socket says 172.21.15.121:51426) 2024-03-29T11:48:18.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.552+0000 7f6013fff640 1 -- 172.21.15.121:0/3988436977 learned_addr learned my addr 172.21.15.121:0/3988436977 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-29T11:48:18.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.552+0000 7f6010ff9640 10 client.?.objecter ms_handle_connect 0x7f601409ac40 2024-03-29T11:48:18.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.552+0000 7f6010ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:18.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.552+0000 7f6010ff9640 10 client.?.objecter ms_handle_connect 0x7f60140ad140 2024-03-29T11:48:18.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.552+0000 7f6010ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:18.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.552+0000 7f6010ff9640 10 client.?.objecter ms_handle_connect 0x7f60141d77e0 2024-03-29T11:48:18.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.552+0000 7f6010ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:18.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.552+0000 7f6010ff9640 1 -- 172.21.15.121:0/3988436977 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3059479763 0 0) 0x7f60141d7db0 con 0x7f601409ac40 2024-03-29T11:48:18.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.552+0000 7f6010ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:48:18.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.552+0000 7f6010ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:18.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.552+0000 7f6010ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:18.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.552+0000 7f6010ff9640 10 monclient(hunting): my global_id is 4120 2024-03-29T11:48:18.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.552+0000 7f6010ff9640 1 -- 172.21.15.121:0/3988436977 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6000004eb0 con 0x7f601409ac40 2024-03-29T11:48:18.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 1 -- 172.21.15.121:0/3988436977 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4281402225 0 0) 0x7f60141d7f40 con 0x7f60140ad140 2024-03-29T11:48:18.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:48:18.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:18.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:18.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 10 monclient(hunting): my global_id is 4112 2024-03-29T11:48:18.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 1 -- 172.21.15.121:0/3988436977 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60141d7db0 con 0x7f60140ad140 2024-03-29T11:48:18.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 1 -- 172.21.15.121:0/3988436977 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2647681758 0 0) 0x7f60141d7c20 con 0x7f60141d77e0 2024-03-29T11:48:18.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:48:18.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:18.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:18.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 10 monclient(hunting): my global_id is 4113 2024-03-29T11:48:18.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 1 -- 172.21.15.121:0/3988436977 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60141d7f40 con 0x7f60141d77e0 2024-03-29T11:48:18.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 1 -- 172.21.15.121:0/3988436977 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2945209088 0 0) 0x7f6000004eb0 con 0x7f601409ac40 2024-03-29T11:48:18.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 1 -- 172.21.15.121:0/3988436977 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f60141d7c20 con 0x7f601409ac40 2024-03-29T11:48:18.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 1 -- 172.21.15.121:0/3988436977 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f6004001b40 con 0x7f601409ac40 2024-03-29T11:48:18.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:18.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:48:18.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 10 monclient(hunting): dump: 2024-03-29T11:48:18.589 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:18.589 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:18.589 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:18.589 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:18.589 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:18.589 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:18.590 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:18.590 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:18.590 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:18.590 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:18.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:48:18.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 1 -- 172.21.15.121:0/3988436977 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3817851730 0 0) 0x7f60141d7db0 con 0x7f60140ad140 2024-03-29T11:48:18.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 1 -- 172.21.15.121:0/3988436977 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6000004eb0 con 0x7f60140ad140 2024-03-29T11:48:18.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 1 -- 172.21.15.121:0/3988436977 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f6008001860 con 0x7f60140ad140 2024-03-29T11:48:18.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:18.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:48:18.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 10 monclient(hunting): dump: 2024-03-29T11:48:18.592 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:18.592 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:18.592 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:18.592 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:18.592 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:18.592 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:18.593 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:18.593 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:18.593 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:18.593 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:18.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 1 -- 172.21.15.121:0/3988436977 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1944287831 0 0) 0x7f60141d7f40 con 0x7f60141d77e0 2024-03-29T11:48:18.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 1 -- 172.21.15.121:0/3988436977 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f60141d7db0 con 0x7f60141d77e0 2024-03-29T11:48:18.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 1 -- 172.21.15.121:0/3988436977 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f600c001a40 con 0x7f60141d77e0 2024-03-29T11:48:18.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:18.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:48:18.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 10 monclient(hunting): dump: 2024-03-29T11:48:18.594 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:18.594 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:18.595 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:18.595 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:18.595 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:18.595 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:18.595 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:18.595 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:18.596 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:18.596 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:18.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 1 -- 172.21.15.121:0/3988436977 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4115297550 0 0) 0x7f60141d7c20 con 0x7f601409ac40 2024-03-29T11:48:18.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 1 -- 172.21.15.121:0/3988436977 >> v1:172.21.15.98:6790/0 conn(0x7f60140ad140 legacy=0x7f60140afc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:18.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 1 -- 172.21.15.121:0/3988436977 >> v1:172.21.15.98:6789/0 conn(0x7f60141d77e0 legacy=0x7f60141e1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:18.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:48:18.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 1 monclient: found mon.b 2024-03-29T11:48:18.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:18.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:18.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 1 -- 172.21.15.121:0/3988436977 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60141e6390 con 0x7f601409ac40 2024-03-29T11:48:18.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 10 monclient: _finish_auth 0 2024-03-29T11:48:18.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:48:18.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:18.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f601a9e6640 5 monclient: authenticate success, global_id 4120 2024-03-29T11:48:18.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f601a9e6640 10 monclient: _renew_subs 2024-03-29T11:48:18.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f601a9e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:18.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f601a9e6640 1 -- 172.21.15.121:0/3988436977 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60141e3360 con 0x7f601409ac40 2024-03-29T11:48:18.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f601a9e6640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:48:18.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f601a9e6640 10 monclient: _renew_subs 2024-03-29T11:48:18.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f601a9e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:18.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f601a9e6640 1 -- 172.21.15.121:0/3988436977 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f60141e3890 con 0x7f601409ac40 2024-03-29T11:48:18.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f601a9e6640 1 librados: init done 2024-03-29T11:48:18.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 1 -- 172.21.15.121:0/3988436977 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6004004a90 con 0x7f601409ac40 2024-03-29T11:48:18.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:18.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 1 -- 172.21.15.121:0/3988436977 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f6004004e20 con 0x7f601409ac40 2024-03-29T11:48:18.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:18.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f60127fc640 4 set_mon_vals no callback set 2024-03-29T11:48:18.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:48:18.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 10 monclient: dump: 2024-03-29T11:48:18.601 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:18.601 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:18.601 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:18.601 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:18.601 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:18.601 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:18.602 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:18.602 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:18.602 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:18.602 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:18.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 1 -- 172.21.15.121:0/3988436977 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6004005390 con 0x7f601409ac40 2024-03-29T11:48:18.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 1 -- 172.21.15.121:0/3988436977 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 981920929 0 0) 0x7f6004006610 con 0x7f601409ac40 2024-03-29T11:48:18.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 10 client.4120.objecter ms_dispatch 0x7f6014000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T11:48:18.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 3 client.4120.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T11:48:18.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 3 client.4120.objecter handle_osd_map decoding full epoch 2 2024-03-29T11:48:18.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.553+0000 7f6010ff9640 20 client.4120.objecter dump_active .. 0 homeless 2024-03-29T11:48:18.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.554+0000 7f601a9e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:18.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.554+0000 7f601a9e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:18.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.554+0000 7f601a9e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:18.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.554+0000 7f601a9e6640 1 -- 172.21.15.121:0/3988436977 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ff4005080 con 0x7f601409ac40 2024-03-29T11:48:18.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.558+0000 7f6010ff9640 1 -- 172.21.15.121:0/3988436977 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6004006b60 con 0x7f601409ac40 2024-03-29T11:48:18.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.558+0000 7f6010ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:18.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.558+0000 7f6010ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:48:18.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.629+0000 7f601a9e6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c3764135-6315-4c21-b430-e3515422b4a5", "id": 0}] 2024-03-29T11:48:18.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.629+0000 7f601a9e6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c3764135-6315-4c21-b430-e3515422b4a5", "id": 0}] 2024-03-29T11:48:18.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.629+0000 7f601a9e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:18.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.629+0000 7f601a9e6640 1 -- 172.21.15.121:0/3988436977 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd new", "uuid": "c3764135-6315-4c21-b430-e3515422b4a5", "id": 0} v 0) v1 -- 0x7f5ff40058b0 con 0x7f601409ac40 2024-03-29T11:48:18.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.634+0000 7f6010ff9640 1 -- 172.21.15.121:0/3988436977 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c3764135-6315-4c21-b430-e3515422b4a5", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 270205591 0 732707515) 0x7f6004005820 con 0x7f601409ac40 2024-03-29T11:48:18.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.634+0000 7f6010ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c3764135-6315-4c21-b430-e3515422b4a5", "id": 0}] 2024-03-29T11:48:18.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.634+0000 7f6010ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:48:18.640 INFO:teuthology.orchestra.run.smithi121.stdout:0 2024-03-29T11:48:18.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.635+0000 7f601a9e6640 10 librados: watch_flush enter 2024-03-29T11:48:18.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.635+0000 7f601a9e6640 10 librados: watch_flush exit 2024-03-29T11:48:18.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.635+0000 7f601a9e6640 20 client.4120.objecter shutdown clearing up homeless session... 2024-03-29T11:48:18.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.635+0000 7f601a9e6640 10 client.4120.objecter successfully canceled tick 2024-03-29T11:48:18.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.635+0000 7f601a9e6640 10 monclient: shutdown 2024-03-29T11:48:18.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.635+0000 7f601a9e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:18.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.635+0000 7f601a9e6640 1 -- 172.21.15.121:0/3988436977 >> v1:172.21.15.121:6789/0 conn(0x7f601409ac40 legacy=0x7f60140b09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:18.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.635+0000 7f601a9e6640 1 -- 172.21.15.121:0/3988436977 shutdown_connections 2024-03-29T11:48:18.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.635+0000 7f601a9e6640 1 -- 172.21.15.121:0/3988436977 >> 172.21.15.121:0/3988436977 conn(0x7f6014066b00 msgr2=0x7f601413d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:18.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.635+0000 7f601a9e6640 1 -- 172.21.15.121:0/3988436977 shutdown_connections 2024-03-29T11:48:18.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.636+0000 7f601a9e6640 1 -- 172.21.15.121:0/3988436977 wait complete. 2024-03-29T11:48:18.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.636+0000 7f601a9e6640 1 librados: shutdown 2024-03-29T11:48:18.651 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new c3a0e291-f5e3-4542-b78b-003929905548 1 2024-03-29T11:48:18.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.745+0000 7f94b358f640 10 monclient: get_monmap_and_config 2024-03-29T11:48:18.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.745+0000 7f94b358f640 10 monclient: build_initial_monmap 2024-03-29T11:48:18.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.745+0000 7f94b358f640 10 monclient: monmap: 2024-03-29T11:48:18.751 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-29T11:48:18.751 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:18.752 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:48:18.746710+0000 2024-03-29T11:48:18.752 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:48:18.746710+0000 2024-03-29T11:48:18.752 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:18.752 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:18.752 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:48:18.752 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:48:18.752 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:48:18.752 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:18.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.746+0000 7f94b358f640 1 Processor -- start 2024-03-29T11:48:18.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.746+0000 7f94b358f640 1 -- start start 2024-03-29T11:48:18.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.746+0000 7f94b358f640 10 monclient: init 2024-03-29T11:48:18.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.746+0000 7f94b358f640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:18.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.746+0000 7f94b358f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:48:18.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.746+0000 7f94b358f640 10 monclient(hunting): picked mon.noname-a con 0x7f94ac09ac40 addr v1:172.21.15.98:6789/0 2024-03-29T11:48:18.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.746+0000 7f94b358f640 10 monclient(hunting): picked mon.noname-c con 0x7f94ac09c870 addr v1:172.21.15.121:6789/0 2024-03-29T11:48:18.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.747+0000 7f94b358f640 10 monclient(hunting): picked mon.noname-b con 0x7f94ac0b4360 addr v1:172.21.15.98:6790/0 2024-03-29T11:48:18.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.747+0000 7f94b358f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94ac09dec0 con 0x7f94ac09ac40 2024-03-29T11:48:18.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.747+0000 7f94b358f640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94ac09da30 con 0x7f94ac09c870 2024-03-29T11:48:18.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.747+0000 7f94b358f640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94ac0b39c0 con 0x7f94ac0b4360 2024-03-29T11:48:18.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.747+0000 7f94b358f640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:18.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.747+0000 7f94b358f640 10 monclient(hunting): authenticate will time out at 808.917305s 2024-03-29T11:48:18.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.747+0000 7f94b0b2a640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f94ac09c870 0x7f94ac13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:51436/0 (socket says 172.21.15.121:51436) 2024-03-29T11:48:18.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.747+0000 7f94b0b2a640 1 -- 172.21.15.121:0/2282747793 learned_addr learned my addr 172.21.15.121:0/2282747793 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-29T11:48:18.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.747+0000 7f94a3fff640 1 -- 172.21.15.121:0/2282747793 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4081795407 0 0) 0x7f94ac09da30 con 0x7f94ac09c870 2024-03-29T11:48:18.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.747+0000 7f94a3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:48:18.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.747+0000 7f94a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:18.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.747+0000 7f94a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:18.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.747+0000 7f94a3fff640 10 monclient(hunting): my global_id is 4123 2024-03-29T11:48:18.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.747+0000 7f94a3fff640 1 -- 172.21.15.121:0/2282747793 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9498003520 con 0x7f94ac09c870 2024-03-29T11:48:18.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.747+0000 7f94a3fff640 1 -- 172.21.15.121:0/2282747793 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1538492395 0 0) 0x7f9498003520 con 0x7f94ac09c870 2024-03-29T11:48:18.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.747+0000 7f94a3fff640 1 -- 172.21.15.121:0/2282747793 >> v1:172.21.15.98:6790/0 conn(0x7f94ac0b4360 legacy=0x7f94ac14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:18.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.747+0000 7f94a3fff640 1 -- 172.21.15.121:0/2282747793 >> v1:172.21.15.98:6789/0 conn(0x7f94ac09ac40 legacy=0x7f94ac0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:18.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.747+0000 7f94a3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:48:18.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.747+0000 7f94a3fff640 1 monclient: found mon.noname-c 2024-03-29T11:48:18.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.747+0000 7f94a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:18.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.747+0000 7f94a3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:48:18.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.747+0000 7f94a3fff640 1 -- 172.21.15.121:0/2282747793 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94ac0b5160 con 0x7f94ac09c870 2024-03-29T11:48:18.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.747+0000 7f94a3fff640 10 monclient: _finish_auth 0 2024-03-29T11:48:18.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.747+0000 7f94a3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:48:18.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.747+0000 7f94a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:18.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.747+0000 7f94b358f640 5 monclient: authenticate success, global_id 4123 2024-03-29T11:48:18.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.747+0000 7f94a3fff640 1 -- 172.21.15.121:0/2282747793 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f949c001690 con 0x7f94ac09c870 2024-03-29T11:48:18.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.747+0000 7f94a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:18.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.747+0000 7f94a3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:48:18.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.747+0000 7f94a3fff640 10 monclient: dump: 2024-03-29T11:48:18.763 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:18.763 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:18.763 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:18.763 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:18.763 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:18.763 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:18.763 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:18.763 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:18.763 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:18.763 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:18.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.747+0000 7f94b358f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:48:18.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.747+0000 7f94a3fff640 1 -- 172.21.15.121:0/2282747793 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f949c001870 con 0x7f94ac09c870 2024-03-29T11:48:18.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.747+0000 7f94a3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:18.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.747+0000 7f94a3fff640 1 -- 172.21.15.121:0/2282747793 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f949c001c80 con 0x7f94ac09c870 2024-03-29T11:48:18.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.747+0000 7f94b358f640 10 monclient: get_monmap_and_config success 2024-03-29T11:48:18.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.747+0000 7f94b358f640 4 set_mon_vals no callback set 2024-03-29T11:48:18.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.747+0000 7f94a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:18.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.747+0000 7f94a3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:48:18.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.747+0000 7f94a3fff640 10 monclient: dump: 2024-03-29T11:48:18.764 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:18.764 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:18.764 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:18.764 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:18.764 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:18.764 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:18.764 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:18.764 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:18.764 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:18.764 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:18.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.748+0000 7f94b358f640 10 monclient: shutdown 2024-03-29T11:48:18.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.748+0000 7f94b358f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:18.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.748+0000 7f94b358f640 1 -- 172.21.15.121:0/2282747793 >> v1:172.21.15.121:6789/0 conn(0x7f94ac09c870 legacy=0x7f94ac13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:18.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.748+0000 7f94b358f640 1 -- 172.21.15.121:0/2282747793 shutdown_connections 2024-03-29T11:48:18.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.748+0000 7f94b358f640 1 -- 172.21.15.121:0/2282747793 >> 172.21.15.121:0/2282747793 conn(0x7f94ac066b00 msgr2=0x7f94ac066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:18.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.748+0000 7f94b358f640 1 -- 172.21.15.121:0/2282747793 shutdown_connections 2024-03-29T11:48:18.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.748+0000 7f94b358f640 1 -- 172.21.15.121:0/2282747793 wait complete. 2024-03-29T11:48:18.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.748+0000 7f94b358f640 10 monclient: build_initial_monmap 2024-03-29T11:48:18.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.748+0000 7f94b358f640 10 monclient: monmap: 2024-03-29T11:48:18.765 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-29T11:48:18.765 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:48:18.765 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-03-29T11:48:18.765 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-03-29T11:48:18.765 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:18.765 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:18.765 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:48:18.765 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:48:18.766 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:48:18.766 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:18.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.749+0000 7f94b358f640 1 librados: starting msgr at 2024-03-29T11:48:18.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.749+0000 7f94b358f640 1 librados: starting objecter 2024-03-29T11:48:18.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.749+0000 7f94b358f640 1 Processor -- start 2024-03-29T11:48:18.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.749+0000 7f94b358f640 1 -- start start 2024-03-29T11:48:18.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.749+0000 7f94b358f640 1 librados: setting wanted keys 2024-03-29T11:48:18.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.749+0000 7f94b358f640 1 librados: calling monclient init 2024-03-29T11:48:18.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.749+0000 7f94b358f640 10 monclient: init 2024-03-29T11:48:18.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.749+0000 7f94b358f640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:18.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.749+0000 7f94b358f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:48:18.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.749+0000 7f94b358f640 10 monclient(hunting): picked mon.noname-a con 0x7f94ac09ac40 addr v1:172.21.15.98:6789/0 2024-03-29T11:48:18.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.749+0000 7f94b358f640 10 monclient(hunting): picked mon.noname-c con 0x7f94ac1c82b0 addr v1:172.21.15.98:6790/0 2024-03-29T11:48:18.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.749+0000 7f94b358f640 10 monclient(hunting): picked mon.noname-b con 0x7f94ac13a5a0 addr v1:172.21.15.121:6789/0 2024-03-29T11:48:18.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.749+0000 7f94b358f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94ac1ca580 con 0x7f94ac09ac40 2024-03-29T11:48:18.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.749+0000 7f94b358f640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94ac1ca7d0 con 0x7f94ac13a5a0 2024-03-29T11:48:18.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.749+0000 7f94b358f640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94ac1c8740 con 0x7f94ac1c82b0 2024-03-29T11:48:18.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.749+0000 7f94b358f640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:18.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.749+0000 7f94b358f640 10 monclient(hunting): authenticate will time out at 808.920013s 2024-03-29T11:48:18.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.749+0000 7f94b1b2c640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f94ac13a5a0 0x7f94ac1daf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:51450/0 (socket says 172.21.15.121:51450) 2024-03-29T11:48:18.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.749+0000 7f94b1b2c640 1 -- 172.21.15.121:0/2760990054 learned_addr learned my addr 172.21.15.121:0/2760990054 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-29T11:48:18.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.749+0000 7f94a1ffb640 10 client.?.objecter ms_handle_connect 0x7f94ac13a5a0 2024-03-29T11:48:18.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.749+0000 7f94a1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:18.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.749+0000 7f94a1ffb640 10 client.?.objecter ms_handle_connect 0x7f94ac09ac40 2024-03-29T11:48:18.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.749+0000 7f94a1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:18.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.749+0000 7f94a1ffb640 10 client.?.objecter ms_handle_connect 0x7f94ac1c82b0 2024-03-29T11:48:18.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.749+0000 7f94a1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:18.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 1 -- 172.21.15.121:0/2760990054 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3728022140 0 0) 0x7f94ac1ca7d0 con 0x7f94ac13a5a0 2024-03-29T11:48:18.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:48:18.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:18.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:18.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 10 monclient(hunting): my global_id is 4126 2024-03-29T11:48:18.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 1 -- 172.21.15.121:0/2760990054 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9490004eb0 con 0x7f94ac13a5a0 2024-03-29T11:48:18.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 1 -- 172.21.15.121:0/2760990054 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 682937144 0 0) 0x7f94ac1ca580 con 0x7f94ac09ac40 2024-03-29T11:48:18.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:48:18.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:18.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:18.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 10 monclient(hunting): my global_id is 4119 2024-03-29T11:48:18.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 1 -- 172.21.15.121:0/2760990054 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94ac1ca7d0 con 0x7f94ac09ac40 2024-03-29T11:48:18.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 1 -- 172.21.15.121:0/2760990054 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2005742705 0 0) 0x7f94ac1c8740 con 0x7f94ac1c82b0 2024-03-29T11:48:18.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:48:18.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:18.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:18.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 10 monclient(hunting): my global_id is 4118 2024-03-29T11:48:18.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 1 -- 172.21.15.121:0/2760990054 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94ac1ca580 con 0x7f94ac1c82b0 2024-03-29T11:48:18.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 1 -- 172.21.15.121:0/2760990054 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2182749876 0 0) 0x7f9490004eb0 con 0x7f94ac13a5a0 2024-03-29T11:48:18.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 1 -- 172.21.15.121:0/2760990054 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94ac1c8740 con 0x7f94ac13a5a0 2024-03-29T11:48:18.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 1 -- 172.21.15.121:0/2760990054 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f94a4001c30 con 0x7f94ac13a5a0 2024-03-29T11:48:18.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:18.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:48:18.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 10 monclient(hunting): dump: 2024-03-29T11:48:18.769 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:18.769 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:18.769 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:18.769 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:18.769 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:18.769 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:18.769 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:18.769 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:18.770 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:18.770 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:18.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:48:18.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 1 -- 172.21.15.121:0/2760990054 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1941674911 0 0) 0x7f94ac1ca580 con 0x7f94ac1c82b0 2024-03-29T11:48:18.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 1 -- 172.21.15.121:0/2760990054 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9490004eb0 con 0x7f94ac1c82b0 2024-03-29T11:48:18.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 1 -- 172.21.15.121:0/2760990054 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3998163492 0 0) 0x7f94ac1ca7d0 con 0x7f94ac09ac40 2024-03-29T11:48:18.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 1 -- 172.21.15.121:0/2760990054 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94ac1ca580 con 0x7f94ac09ac40 2024-03-29T11:48:18.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 1 -- 172.21.15.121:0/2760990054 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f949c0019e0 con 0x7f94ac1c82b0 2024-03-29T11:48:18.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:18.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:48:18.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 10 monclient(hunting): dump: 2024-03-29T11:48:18.770 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:18.770 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:18.770 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:18.770 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:18.771 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:18.771 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:18.771 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:18.771 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:18.771 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:18.771 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:18.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 1 -- 172.21.15.121:0/2760990054 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f94a8001710 con 0x7f94ac09ac40 2024-03-29T11:48:18.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:18.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:48:18.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 10 monclient(hunting): dump: 2024-03-29T11:48:18.771 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:18.771 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:18.771 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:18.771 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:18.771 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:18.772 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:18.772 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:18.772 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:18.772 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:18.772 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:18.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 1 -- 172.21.15.121:0/2760990054 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3272806757 0 0) 0x7f94ac1c8740 con 0x7f94ac13a5a0 2024-03-29T11:48:18.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 1 -- 172.21.15.121:0/2760990054 >> v1:172.21.15.98:6790/0 conn(0x7f94ac1c82b0 legacy=0x7f94ac1da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:18.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 1 -- 172.21.15.121:0/2760990054 >> v1:172.21.15.98:6789/0 conn(0x7f94ac09ac40 legacy=0x7f94ac139e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:18.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:48:18.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 1 monclient: found mon.b 2024-03-29T11:48:18.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:18.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:18.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 1 -- 172.21.15.121:0/2760990054 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94ac1c9420 con 0x7f94ac13a5a0 2024-03-29T11:48:18.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:48:18.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:48:18.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:18.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94b358f640 5 monclient: authenticate success, global_id 4126 2024-03-29T11:48:18.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94b358f640 10 monclient: _renew_subs 2024-03-29T11:48:18.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94b358f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:18.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94b358f640 1 -- 172.21.15.121:0/2760990054 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94ac1c9670 con 0x7f94ac13a5a0 2024-03-29T11:48:18.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 1 -- 172.21.15.121:0/2760990054 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f94a4003990 con 0x7f94ac13a5a0 2024-03-29T11:48:18.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:18.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 1 -- 172.21.15.121:0/2760990054 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f94a4003d20 con 0x7f94ac13a5a0 2024-03-29T11:48:18.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:18.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:48:18.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 10 monclient: dump: 2024-03-29T11:48:18.773 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:18.773 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:18.773 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:18.773 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:18.774 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:18.774 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:18.774 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:18.774 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:18.774 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:18.774 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:18.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 1 -- 172.21.15.121:0/2760990054 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f94a40042d0 con 0x7f94ac13a5a0 2024-03-29T11:48:18.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94b358f640 10 client.4126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:48:18.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94b358f640 10 monclient: _renew_subs 2024-03-29T11:48:18.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a37fe640 4 set_mon_vals no callback set 2024-03-29T11:48:18.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94b358f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:18.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94b358f640 1 -- 172.21.15.121:0/2760990054 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f94ac1cb140 con 0x7f94ac13a5a0 2024-03-29T11:48:18.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94b358f640 1 librados: init done 2024-03-29T11:48:18.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 1 -- 172.21.15.121:0/2760990054 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 3299404752 0 0) 0x7f94a4005b40 con 0x7f94ac13a5a0 2024-03-29T11:48:18.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 10 client.4126.objecter ms_dispatch 0x7f94ac000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T11:48:18.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 3 client.4126.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T11:48:18.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 3 client.4126.objecter handle_osd_map decoding full epoch 3 2024-03-29T11:48:18.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.750+0000 7f94a1ffb640 20 client.4126.objecter dump_active .. 0 homeless 2024-03-29T11:48:18.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.751+0000 7f94b358f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:18.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.751+0000 7f94b358f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:18.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.751+0000 7f94b358f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:18.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.751+0000 7f94b358f640 1 -- 172.21.15.121:0/2760990054 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94ac13e2c0 con 0x7f94ac13a5a0 2024-03-29T11:48:18.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.753+0000 7f94a1ffb640 1 -- 172.21.15.121:0/2760990054 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f94a4005e70 con 0x7f94ac13a5a0 2024-03-29T11:48:18.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.753+0000 7f94a1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:18.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.753+0000 7f94a1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:48:18.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.826+0000 7f94b358f640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c3a0e291-f5e3-4542-b78b-003929905548", "id": 1}] 2024-03-29T11:48:18.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.826+0000 7f94b358f640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c3a0e291-f5e3-4542-b78b-003929905548", "id": 1}] 2024-03-29T11:48:18.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.826+0000 7f94b358f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:18.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.826+0000 7f94b358f640 1 -- 172.21.15.121:0/2760990054 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd new", "uuid": "c3a0e291-f5e3-4542-b78b-003929905548", "id": 1} v 0) v1 -- 0x7f94ac09b020 con 0x7f94ac13a5a0 2024-03-29T11:48:18.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.834+0000 7f94a1ffb640 1 -- 172.21.15.121:0/2760990054 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c3a0e291-f5e3-4542-b78b-003929905548", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 288939134 0 940483276) 0x7f94a40069f0 con 0x7f94ac13a5a0 2024-03-29T11:48:18.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.834+0000 7f94a1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c3a0e291-f5e3-4542-b78b-003929905548", "id": 1}] 2024-03-29T11:48:18.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.834+0000 7f94a1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:48:18.840 INFO:teuthology.orchestra.run.smithi121.stdout:1 2024-03-29T11:48:18.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.834+0000 7f94b358f640 10 librados: watch_flush enter 2024-03-29T11:48:18.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.835+0000 7f94b358f640 10 librados: watch_flush exit 2024-03-29T11:48:18.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.835+0000 7f94b358f640 20 client.4126.objecter shutdown clearing up homeless session... 2024-03-29T11:48:18.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.835+0000 7f94b358f640 10 client.4126.objecter successfully canceled tick 2024-03-29T11:48:18.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.835+0000 7f94b358f640 10 monclient: shutdown 2024-03-29T11:48:18.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.835+0000 7f94b358f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:18.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.835+0000 7f94b358f640 1 -- 172.21.15.121:0/2760990054 >> v1:172.21.15.121:6789/0 conn(0x7f94ac13a5a0 legacy=0x7f94ac1daf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:18.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.835+0000 7f94b358f640 1 -- 172.21.15.121:0/2760990054 shutdown_connections 2024-03-29T11:48:18.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.835+0000 7f94b358f640 1 -- 172.21.15.121:0/2760990054 >> 172.21.15.121:0/2760990054 conn(0x7f94ac066b00 msgr2=0x7f94ac0b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:18.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.835+0000 7f94b358f640 1 -- 172.21.15.121:0/2760990054 shutdown_connections 2024-03-29T11:48:18.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.835+0000 7f94b358f640 1 -- 172.21.15.121:0/2760990054 wait complete. 2024-03-29T11:48:18.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.835+0000 7f94b358f640 1 librados: shutdown 2024-03-29T11:48:18.851 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new c3d2484d-a574-4ad6-8210-6b4ff2adde91 2 2024-03-29T11:48:18.936 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T11:48:18.931+0000 7faaea042040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:48:18.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.945+0000 7f43614a8640 10 monclient: get_monmap_and_config 2024-03-29T11:48:18.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.945+0000 7f43614a8640 10 monclient: build_initial_monmap 2024-03-29T11:48:18.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.945+0000 7f43614a8640 10 monclient: monmap: 2024-03-29T11:48:18.951 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-29T11:48:18.951 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:18.951 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:48:18.946908+0000 2024-03-29T11:48:18.951 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:48:18.946908+0000 2024-03-29T11:48:18.952 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:18.952 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:18.952 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:48:18.952 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:48:18.952 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:48:18.952 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:18.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.946+0000 7f43614a8640 1 Processor -- start 2024-03-29T11:48:18.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.946+0000 7f43614a8640 1 -- start start 2024-03-29T11:48:18.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.946+0000 7f43614a8640 10 monclient: init 2024-03-29T11:48:18.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f43614a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:18.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f43614a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:48:18.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f43614a8640 10 monclient(hunting): picked mon.noname-b con 0x7f435c066b90 addr v1:172.21.15.98:6790/0 2024-03-29T11:48:18.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f43614a8640 10 monclient(hunting): picked mon.noname-a con 0x7f435c0b0080 addr v1:172.21.15.98:6789/0 2024-03-29T11:48:18.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f43614a8640 10 monclient(hunting): picked mon.noname-c con 0x7f435c072790 addr v1:172.21.15.121:6789/0 2024-03-29T11:48:18.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f43614a8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f435c0afbf0 con 0x7f435c0b0080 2024-03-29T11:48:18.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f43614a8640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f435c0aed90 con 0x7f435c072790 2024-03-29T11:48:18.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f43614a8640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f435c13fbe0 con 0x7f435c066b90 2024-03-29T11:48:18.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f43614a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:18.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f43614a8640 10 monclient(hunting): authenticate will time out at 809.117431s 2024-03-29T11:48:18.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f435b7fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f435c072790 0x7f435c14d9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:51452/0 (socket says 172.21.15.121:51452) 2024-03-29T11:48:18.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f435b7fe640 1 -- 172.21.15.121:0/2200362082 learned_addr learned my addr 172.21.15.121:0/2200362082 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-29T11:48:18.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f4359ffb640 1 -- 172.21.15.121:0/2200362082 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3975464812 0 0) 0x7f435c0aed90 con 0x7f435c072790 2024-03-29T11:48:18.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f4359ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:48:18.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f4359ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:18.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f4359ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:18.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f4359ffb640 10 monclient(hunting): my global_id is 4129 2024-03-29T11:48:18.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f4359ffb640 1 -- 172.21.15.121:0/2200362082 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4344003520 con 0x7f435c072790 2024-03-29T11:48:18.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f4359ffb640 1 -- 172.21.15.121:0/2200362082 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 344541051 0 0) 0x7f435c13fbe0 con 0x7f435c066b90 2024-03-29T11:48:18.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f4359ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:48:18.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f4359ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:18.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f4359ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:18.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f4359ffb640 10 monclient(hunting): my global_id is 4121 2024-03-29T11:48:18.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f4359ffb640 1 -- 172.21.15.121:0/2200362082 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f435c0aed90 con 0x7f435c066b90 2024-03-29T11:48:18.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f4359ffb640 1 -- 172.21.15.121:0/2200362082 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3723139928 0 0) 0x7f435c0afbf0 con 0x7f435c0b0080 2024-03-29T11:48:18.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f4359ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:48:18.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f4359ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:18.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f4359ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:18.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f4359ffb640 10 monclient(hunting): my global_id is 4122 2024-03-29T11:48:18.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f4359ffb640 1 -- 172.21.15.121:0/2200362082 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f435c13fbe0 con 0x7f435c0b0080 2024-03-29T11:48:18.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f4359ffb640 1 -- 172.21.15.121:0/2200362082 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2724257992 0 0) 0x7f4344003520 con 0x7f435c072790 2024-03-29T11:48:18.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f4359ffb640 1 -- 172.21.15.121:0/2200362082 >> v1:172.21.15.98:6790/0 conn(0x7f435c066b90 legacy=0x7f435c13f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:18.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f4359ffb640 1 -- 172.21.15.121:0/2200362082 >> v1:172.21.15.98:6789/0 conn(0x7f435c0b0080 legacy=0x7f435c142140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:18.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f4359ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:48:18.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f4359ffb640 1 monclient: found mon.noname-c 2024-03-29T11:48:18.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f4359ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:18.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f4359ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:48:18.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f4359ffb640 1 -- 172.21.15.121:0/2200362082 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f435c13fd70 con 0x7f435c072790 2024-03-29T11:48:18.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f4359ffb640 10 monclient: _finish_auth 0 2024-03-29T11:48:18.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f4359ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:48:18.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f4359ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:18.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f4359ffb640 1 -- 172.21.15.121:0/2200362082 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f4354001a10 con 0x7f435c072790 2024-03-29T11:48:18.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f43614a8640 5 monclient: authenticate success, global_id 4129 2024-03-29T11:48:18.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f43614a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:48:18.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f4359ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:18.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f4359ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:48:18.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f4359ffb640 10 monclient: dump: 2024-03-29T11:48:18.964 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:18.964 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:18.964 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:18.964 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:18.964 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:18.964 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:18.965 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:18.965 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:18.965 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:18.965 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:18.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f43614a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:48:18.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f4359ffb640 1 -- 172.21.15.121:0/2200362082 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4354001bf0 con 0x7f435c072790 2024-03-29T11:48:18.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f4359ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:18.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f43614a8640 10 monclient: get_monmap_and_config success 2024-03-29T11:48:18.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f4359ffb640 1 -- 172.21.15.121:0/2200362082 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f4354001f80 con 0x7f435c072790 2024-03-29T11:48:18.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f4359ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:18.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f43614a8640 4 set_mon_vals no callback set 2024-03-29T11:48:18.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f4359ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:48:18.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.947+0000 7f4359ffb640 10 monclient: dump: 2024-03-29T11:48:18.965 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:18.965 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:18.966 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:18.966 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:18.966 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:18.966 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:18.966 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:18.966 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:18.966 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:18.966 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:18.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.948+0000 7f43614a8640 10 monclient: shutdown 2024-03-29T11:48:18.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.948+0000 7f43614a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:18.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.948+0000 7f43614a8640 1 -- 172.21.15.121:0/2200362082 >> v1:172.21.15.121:6789/0 conn(0x7f435c072790 legacy=0x7f435c14d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:18.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.948+0000 7f43614a8640 1 -- 172.21.15.121:0/2200362082 shutdown_connections 2024-03-29T11:48:18.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.948+0000 7f43614a8640 1 -- 172.21.15.121:0/2200362082 >> 172.21.15.121:0/2200362082 conn(0x7f435c06b600 msgr2=0x7f435c0b49e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:18.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.948+0000 7f43614a8640 1 -- 172.21.15.121:0/2200362082 shutdown_connections 2024-03-29T11:48:18.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.948+0000 7f43614a8640 1 -- 172.21.15.121:0/2200362082 wait complete. 2024-03-29T11:48:18.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.949+0000 7f43614a8640 10 monclient: build_initial_monmap 2024-03-29T11:48:18.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.949+0000 7f43614a8640 10 monclient: monmap: 2024-03-29T11:48:18.967 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-29T11:48:18.967 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:48:18.967 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-03-29T11:48:18.967 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-03-29T11:48:18.967 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:18.967 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:18.967 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:48:18.967 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:48:18.967 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:48:18.967 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:18.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.949+0000 7f43614a8640 1 librados: starting msgr at 2024-03-29T11:48:18.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.949+0000 7f43614a8640 1 librados: starting objecter 2024-03-29T11:48:18.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.949+0000 7f43614a8640 1 Processor -- start 2024-03-29T11:48:18.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.949+0000 7f43614a8640 1 -- start start 2024-03-29T11:48:18.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.949+0000 7f43614a8640 1 librados: setting wanted keys 2024-03-29T11:48:18.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.949+0000 7f43614a8640 1 librados: calling monclient init 2024-03-29T11:48:18.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.949+0000 7f43614a8640 10 monclient: init 2024-03-29T11:48:18.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43614a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:18.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43614a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:48:18.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43614a8640 10 monclient(hunting): picked mon.noname-c con 0x7f435c066b90 addr v1:172.21.15.98:6790/0 2024-03-29T11:48:18.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43614a8640 10 monclient(hunting): picked mon.noname-b con 0x7f435c072790 addr v1:172.21.15.121:6789/0 2024-03-29T11:48:18.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43614a8640 10 monclient(hunting): picked mon.noname-a con 0x7f435c0b0080 addr v1:172.21.15.98:6789/0 2024-03-29T11:48:18.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43614a8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f435c1d6e90 con 0x7f435c0b0080 2024-03-29T11:48:18.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43614a8640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f435c1d7170 con 0x7f435c072790 2024-03-29T11:48:18.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43614a8640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f435c1d7d10 con 0x7f435c066b90 2024-03-29T11:48:18.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43614a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:18.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43614a8640 10 monclient(hunting): authenticate will time out at 809.120580s 2024-03-29T11:48:18.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f435a7fc640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f435c072790 0x7f435c1d7600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:51458/0 (socket says 172.21.15.121:51458) 2024-03-29T11:48:18.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f435a7fc640 1 -- 172.21.15.121:0/3395739217 learned_addr learned my addr 172.21.15.121:0/3395739217 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-29T11:48:18.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43437fe640 10 client.?.objecter ms_handle_connect 0x7f435c072790 2024-03-29T11:48:18.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43437fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:18.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43437fe640 10 client.?.objecter ms_handle_connect 0x7f435c066b90 2024-03-29T11:48:18.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43437fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:18.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43437fe640 10 client.?.objecter ms_handle_connect 0x7f435c0b0080 2024-03-29T11:48:18.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43437fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:18.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43437fe640 1 -- 172.21.15.121:0/3395739217 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 616234905 0 0) 0x7f435c1d7170 con 0x7f435c072790 2024-03-29T11:48:18.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43437fe640 10 monclient(hunting): _negotiate 2024-03-29T11:48:18.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43437fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:18.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:18.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43437fe640 10 monclient(hunting): my global_id is 4132 2024-03-29T11:48:18.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43437fe640 1 -- 172.21.15.121:0/3395739217 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4348004eb0 con 0x7f435c072790 2024-03-29T11:48:18.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43437fe640 1 -- 172.21.15.121:0/3395739217 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3376607018 0 0) 0x7f435c1d7d10 con 0x7f435c066b90 2024-03-29T11:48:18.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43437fe640 10 monclient(hunting): _negotiate 2024-03-29T11:48:18.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43437fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:18.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:18.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43437fe640 10 monclient(hunting): my global_id is 4124 2024-03-29T11:48:18.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43437fe640 1 -- 172.21.15.121:0/3395739217 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f435c1d7170 con 0x7f435c066b90 2024-03-29T11:48:18.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43437fe640 1 -- 172.21.15.121:0/3395739217 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1259794862 0 0) 0x7f435c1d6e90 con 0x7f435c0b0080 2024-03-29T11:48:18.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43437fe640 10 monclient(hunting): _negotiate 2024-03-29T11:48:18.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43437fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:18.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:18.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43437fe640 10 monclient(hunting): my global_id is 4125 2024-03-29T11:48:18.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43437fe640 1 -- 172.21.15.121:0/3395739217 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f435c1d7d10 con 0x7f435c0b0080 2024-03-29T11:48:18.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43437fe640 1 -- 172.21.15.121:0/3395739217 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4154618500 0 0) 0x7f435c1d7170 con 0x7f435c066b90 2024-03-29T11:48:18.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43437fe640 1 -- 172.21.15.121:0/3395739217 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f435c1d6e90 con 0x7f435c066b90 2024-03-29T11:48:18.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43437fe640 1 -- 172.21.15.121:0/3395739217 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f4350001930 con 0x7f435c066b90 2024-03-29T11:48:18.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:18.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43437fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:48:18.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43437fe640 10 monclient(hunting): dump: 2024-03-29T11:48:18.971 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:18.971 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:18.971 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:18.971 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:18.971 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:18.971 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:18.971 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:18.971 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:18.971 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:18.971 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:18.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43437fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:48:18.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43437fe640 1 -- 172.21.15.121:0/3395739217 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4078838495 0 0) 0x7f4348004eb0 con 0x7f435c072790 2024-03-29T11:48:18.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43437fe640 1 -- 172.21.15.121:0/3395739217 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f435c1d7170 con 0x7f435c072790 2024-03-29T11:48:18.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43437fe640 1 -- 172.21.15.121:0/3395739217 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f434c0018f0 con 0x7f435c072790 2024-03-29T11:48:18.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:18.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43437fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:48:18.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43437fe640 10 monclient(hunting): dump: 2024-03-29T11:48:18.972 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:18.972 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:18.972 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:18.972 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:18.972 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:18.972 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:18.972 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:18.972 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:18.972 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:18.972 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:18.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43437fe640 1 -- 172.21.15.121:0/3395739217 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2118171063 0 0) 0x7f435c1d7d10 con 0x7f435c0b0080 2024-03-29T11:48:18.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.950+0000 7f43437fe640 1 -- 172.21.15.121:0/3395739217 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4348004eb0 con 0x7f435c0b0080 2024-03-29T11:48:18.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.951+0000 7f43437fe640 1 -- 172.21.15.121:0/3395739217 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f4354001d60 con 0x7f435c0b0080 2024-03-29T11:48:18.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.951+0000 7f43437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:18.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.951+0000 7f43437fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:48:18.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.951+0000 7f43437fe640 10 monclient(hunting): dump: 2024-03-29T11:48:18.973 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:18.973 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:18.973 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:18.973 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:18.973 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:18.973 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:18.973 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:18.973 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:18.973 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:18.973 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:18.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.951+0000 7f43437fe640 1 -- 172.21.15.121:0/3395739217 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 296960843 0 0) 0x7f435c1d6e90 con 0x7f435c066b90 2024-03-29T11:48:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.951+0000 7f43437fe640 1 -- 172.21.15.121:0/3395739217 >> v1:172.21.15.121:6789/0 conn(0x7f435c072790 legacy=0x7f435c1d7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.951+0000 7f43437fe640 1 -- 172.21.15.121:0/3395739217 >> v1:172.21.15.98:6789/0 conn(0x7f435c0b0080 legacy=0x7f435c145a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.951+0000 7f43437fe640 10 monclient: _finish_hunting 0 2024-03-29T11:48:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.951+0000 7f43437fe640 1 monclient: found mon.c 2024-03-29T11:48:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.951+0000 7f43437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.951+0000 7f43437fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:48:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.951+0000 7f43437fe640 1 -- 172.21.15.121:0/3395739217 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f435c144c60 con 0x7f435c066b90 2024-03-29T11:48:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.951+0000 7f43437fe640 10 monclient: _finish_auth 0 2024-03-29T11:48:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.951+0000 7f43437fe640 10 monclient: _check_auth_tickets 2024-03-29T11:48:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.951+0000 7f43437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.951+0000 7f43614a8640 5 monclient: authenticate success, global_id 4124 2024-03-29T11:48:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.951+0000 7f43614a8640 10 monclient: _renew_subs 2024-03-29T11:48:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.951+0000 7f43614a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:48:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.951+0000 7f43614a8640 1 -- 172.21.15.121:0/3395739217 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f435c146140 con 0x7f435c066b90 2024-03-29T11:48:18.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.951+0000 7f43614a8640 10 client.4124.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:48:18.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.951+0000 7f43614a8640 10 monclient: _renew_subs 2024-03-29T11:48:18.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.951+0000 7f43614a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:48:18.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.951+0000 7f43614a8640 1 -- 172.21.15.121:0/3395739217 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f435c1436b0 con 0x7f435c066b90 2024-03-29T11:48:18.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.951+0000 7f43614a8640 1 librados: init done 2024-03-29T11:48:18.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.951+0000 7f43437fe640 1 -- 172.21.15.121:0/3395739217 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43500035f0 con 0x7f435c066b90 2024-03-29T11:48:18.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.951+0000 7f43437fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:18.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.951+0000 7f43437fe640 1 -- 172.21.15.121:0/3395739217 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f4350003900 con 0x7f435c066b90 2024-03-29T11:48:18.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.951+0000 7f43437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:18.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.951+0000 7f43437fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:48:18.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.951+0000 7f43597fa640 4 set_mon_vals no callback set 2024-03-29T11:48:18.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.951+0000 7f43437fe640 10 monclient: dump: 2024-03-29T11:48:18.975 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:18.975 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:18.975 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:18.975 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:18.976 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:18.976 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:18.976 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:18.976 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:18.976 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:18.976 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:18.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.951+0000 7f43437fe640 1 -- 172.21.15.121:0/3395739217 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4350003df0 con 0x7f435c066b90 2024-03-29T11:48:18.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.951+0000 7f43437fe640 1 -- 172.21.15.121:0/3395739217 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 3864008110 0 0) 0x7f43500044c0 con 0x7f435c066b90 2024-03-29T11:48:18.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.951+0000 7f43437fe640 10 client.4124.objecter ms_dispatch 0x7f435c000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T11:48:18.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.951+0000 7f43437fe640 3 client.4124.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T11:48:18.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.951+0000 7f43437fe640 3 client.4124.objecter handle_osd_map decoding full epoch 4 2024-03-29T11:48:18.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.951+0000 7f43437fe640 20 client.4124.objecter dump_active .. 0 homeless 2024-03-29T11:48:18.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.952+0000 7f43614a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:18.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.952+0000 7f43614a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:18.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.952+0000 7f43614a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:48:18.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.952+0000 7f43614a8640 1 -- 172.21.15.121:0/3395739217 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f433c005080 con 0x7f435c066b90 2024-03-29T11:48:18.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.954+0000 7f43437fe640 1 -- 172.21.15.121:0/3395739217 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f43500040a0 con 0x7f435c066b90 2024-03-29T11:48:18.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.954+0000 7f43437fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:18.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:18.954+0000 7f43437fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:48:18.991 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T11:48:18.986+0000 7faaea042040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:48:18.997 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:48:18.991+0000 7f4e33c10040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:48:19.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.027+0000 7f43614a8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c3d2484d-a574-4ad6-8210-6b4ff2adde91", "id": 2}] 2024-03-29T11:48:19.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.027+0000 7f43614a8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c3d2484d-a574-4ad6-8210-6b4ff2adde91", "id": 2}] 2024-03-29T11:48:19.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.027+0000 7f43614a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:48:19.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.027+0000 7f43614a8640 1 -- 172.21.15.121:0/3395739217 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd new", "uuid": "c3d2484d-a574-4ad6-8210-6b4ff2adde91", "id": 2} v 0) v1 -- 0x7f433c0058b0 con 0x7f435c066b90 2024-03-29T11:48:19.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.030+0000 7f43437fe640 1 -- 172.21.15.121:0/3395739217 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c3d2484d-a574-4ad6-8210-6b4ff2adde91", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 4174681811 0 216599125) 0x7f4350004740 con 0x7f435c066b90 2024-03-29T11:48:19.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.030+0000 7f43437fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c3d2484d-a574-4ad6-8210-6b4ff2adde91", "id": 2}] 2024-03-29T11:48:19.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.030+0000 7f43437fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:48:19.036 INFO:teuthology.orchestra.run.smithi121.stdout:2 2024-03-29T11:48:19.036 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T11:48:19.030+0000 7faaea042040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:48:19.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.030+0000 7f43614a8640 10 librados: watch_flush enter 2024-03-29T11:48:19.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.030+0000 7f43614a8640 10 librados: watch_flush exit 2024-03-29T11:48:19.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.030+0000 7f43614a8640 20 client.4124.objecter shutdown clearing up homeless session... 2024-03-29T11:48:19.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.030+0000 7f43614a8640 10 client.4124.objecter successfully canceled tick 2024-03-29T11:48:19.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.031+0000 7f43614a8640 10 monclient: shutdown 2024-03-29T11:48:19.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.031+0000 7f43614a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:19.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.031+0000 7f43614a8640 1 -- 172.21.15.121:0/3395739217 >> v1:172.21.15.98:6790/0 conn(0x7f435c066b90 legacy=0x7f435c071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:19.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.031+0000 7f43614a8640 1 -- 172.21.15.121:0/3395739217 shutdown_connections 2024-03-29T11:48:19.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.031+0000 7f43614a8640 1 -- 172.21.15.121:0/3395739217 >> 172.21.15.121:0/3395739217 conn(0x7f435c06b600 msgr2=0x7f435c139050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:19.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.031+0000 7f43614a8640 1 -- 172.21.15.121:0/3395739217 shutdown_connections 2024-03-29T11:48:19.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.031+0000 7f43614a8640 1 -- 172.21.15.121:0/3395739217 wait complete. 2024-03-29T11:48:19.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.031+0000 7f43614a8640 1 librados: shutdown 2024-03-29T11:48:19.047 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new c6c10299-76b6-4de5-ac71-6de9c36b6de4 3 2024-03-29T11:48:19.053 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:48:19.048+0000 7f4e33c10040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:48:19.089 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T11:48:19.084+0000 7faaea042040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:48:19.101 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:48:19.095+0000 7f4e33c10040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:48:19.131 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T11:48:19.126+0000 7faaea042040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:48:19.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.140+0000 7f5af3dcf640 10 monclient: get_monmap_and_config 2024-03-29T11:48:19.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.140+0000 7f5af3dcf640 10 monclient: build_initial_monmap 2024-03-29T11:48:19.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.141+0000 7f5af3dcf640 10 monclient: monmap: 2024-03-29T11:48:19.146 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-29T11:48:19.147 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:19.147 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:48:19.142045+0000 2024-03-29T11:48:19.147 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:48:19.142045+0000 2024-03-29T11:48:19.147 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:19.147 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:19.147 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:48:19.147 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:48:19.147 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:48:19.147 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:19.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.141+0000 7f5af3dcf640 1 Processor -- start 2024-03-29T11:48:19.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.141+0000 7f5af3dcf640 1 -- start start 2024-03-29T11:48:19.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.141+0000 7f5af3dcf640 10 monclient: init 2024-03-29T11:48:19.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.142+0000 7f5af3dcf640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:19.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.142+0000 7f5af3dcf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:48:19.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.142+0000 7f5af3dcf640 10 monclient(hunting): picked mon.noname-c con 0x7f5aec066b90 addr v1:172.21.15.121:6789/0 2024-03-29T11:48:19.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.142+0000 7f5af3dcf640 10 monclient(hunting): picked mon.noname-b con 0x7f5aec13f340 addr v1:172.21.15.98:6790/0 2024-03-29T11:48:19.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.142+0000 7f5af3dcf640 10 monclient(hunting): picked mon.noname-a con 0x7f5aec13f7a0 addr v1:172.21.15.98:6789/0 2024-03-29T11:48:19.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.142+0000 7f5af3dcf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aec13fc30 con 0x7f5aec13f7a0 2024-03-29T11:48:19.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.142+0000 7f5af3dcf640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aec0afd70 con 0x7f5aec066b90 2024-03-29T11:48:19.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.142+0000 7f5af3dcf640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aec0aec00 con 0x7f5aec13f340 2024-03-29T11:48:19.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.142+0000 7f5af3dcf640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:19.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.142+0000 7f5af3dcf640 10 monclient(hunting): authenticate will time out at 809.312706s 2024-03-29T11:48:19.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.142+0000 7f5af1b6b640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f5aec066b90 0x7f5aec0af580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:51470/0 (socket says 172.21.15.121:51470) 2024-03-29T11:48:19.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.142+0000 7f5af1b6b640 1 -- 172.21.15.121:0/135162364 learned_addr learned my addr 172.21.15.121:0/135162364 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-29T11:48:19.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.142+0000 7f5af0b69640 1 -- 172.21.15.121:0/135162364 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3183746488 0 0) 0x7f5aec0afd70 con 0x7f5aec066b90 2024-03-29T11:48:19.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.142+0000 7f5af0b69640 10 monclient(hunting): _negotiate 2024-03-29T11:48:19.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.142+0000 7f5af0b69640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:19.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.142+0000 7f5af0b69640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:19.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.142+0000 7f5af0b69640 10 monclient(hunting): my global_id is 4135 2024-03-29T11:48:19.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.142+0000 7f5af0b69640 1 -- 172.21.15.121:0/135162364 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ad8003520 con 0x7f5aec066b90 2024-03-29T11:48:19.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.142+0000 7f5af0b69640 1 -- 172.21.15.121:0/135162364 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 338254367 0 0) 0x7f5aec13fc30 con 0x7f5aec13f7a0 2024-03-29T11:48:19.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.142+0000 7f5af0b69640 10 monclient(hunting): _negotiate 2024-03-29T11:48:19.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.142+0000 7f5af0b69640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:19.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.142+0000 7f5af0b69640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:19.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.142+0000 7f5af0b69640 10 monclient(hunting): my global_id is 4128 2024-03-29T11:48:19.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.142+0000 7f5af0b69640 1 -- 172.21.15.121:0/135162364 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5aec0afd70 con 0x7f5aec13f7a0 2024-03-29T11:48:19.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.142+0000 7f5af0b69640 1 -- 172.21.15.121:0/135162364 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 120455944 0 0) 0x7f5aec0aec00 con 0x7f5aec13f340 2024-03-29T11:48:19.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.142+0000 7f5af0b69640 10 monclient(hunting): _negotiate 2024-03-29T11:48:19.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.142+0000 7f5af0b69640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:19.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.142+0000 7f5af0b69640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:19.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.142+0000 7f5af0b69640 10 monclient(hunting): my global_id is 4127 2024-03-29T11:48:19.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.142+0000 7f5af0b69640 1 -- 172.21.15.121:0/135162364 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5aec13fc30 con 0x7f5aec13f340 2024-03-29T11:48:19.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.143+0000 7f5af0b69640 1 -- 172.21.15.121:0/135162364 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1509627810 0 0) 0x7f5ad8003520 con 0x7f5aec066b90 2024-03-29T11:48:19.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.143+0000 7f5af0b69640 1 -- 172.21.15.121:0/135162364 >> v1:172.21.15.98:6790/0 conn(0x7f5aec13f340 legacy=0x7f5aec142060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:19.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.143+0000 7f5af0b69640 1 -- 172.21.15.121:0/135162364 >> v1:172.21.15.98:6789/0 conn(0x7f5aec13f7a0 legacy=0x7f5aec142770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:19.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.143+0000 7f5af0b69640 10 monclient: _finish_hunting 0 2024-03-29T11:48:19.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.143+0000 7f5af0b69640 1 monclient: found mon.noname-c 2024-03-29T11:48:19.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.143+0000 7f5af0b69640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:19.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.143+0000 7f5af0b69640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:48:19.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.143+0000 7f5af0b69640 1 -- 172.21.15.121:0/135162364 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aec0aed90 con 0x7f5aec066b90 2024-03-29T11:48:19.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.143+0000 7f5af0b69640 10 monclient: _finish_auth 0 2024-03-29T11:48:19.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.143+0000 7f5af0b69640 10 monclient: _check_auth_tickets 2024-03-29T11:48:19.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.143+0000 7f5af0b69640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:19.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.143+0000 7f5af0b69640 1 -- 172.21.15.121:0/135162364 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f5adc001690 con 0x7f5aec066b90 2024-03-29T11:48:19.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.143+0000 7f5af0b69640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:19.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.143+0000 7f5af0b69640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:48:19.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.143+0000 7f5af0b69640 10 monclient: dump: 2024-03-29T11:48:19.161 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:19.162 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:19.162 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.162 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.162 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:19.162 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:19.162 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:19.162 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:19.162 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:19.162 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:19.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.143+0000 7f5af3dcf640 5 monclient: authenticate success, global_id 4135 2024-03-29T11:48:19.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.143+0000 7f5af3dcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:48:19.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.143+0000 7f5af0b69640 1 -- 172.21.15.121:0/135162364 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5adc001870 con 0x7f5aec066b90 2024-03-29T11:48:19.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.143+0000 7f5af0b69640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:19.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.143+0000 7f5af0b69640 1 -- 172.21.15.121:0/135162364 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f5adc001c80 con 0x7f5aec066b90 2024-03-29T11:48:19.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.143+0000 7f5af3dcf640 10 monclient: get_monmap_and_config success 2024-03-29T11:48:19.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.143+0000 7f5af3dcf640 4 set_mon_vals no callback set 2024-03-29T11:48:19.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.143+0000 7f5af0b69640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:19.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.143+0000 7f5af0b69640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:48:19.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.143+0000 7f5af0b69640 10 monclient: dump: 2024-03-29T11:48:19.163 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:19.163 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:19.163 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.163 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.163 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:19.163 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:19.163 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:19.163 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:19.163 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:19.163 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:19.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.143+0000 7f5af3dcf640 10 monclient: shutdown 2024-03-29T11:48:19.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.143+0000 7f5af3dcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:19.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.143+0000 7f5af3dcf640 1 -- 172.21.15.121:0/135162364 >> v1:172.21.15.121:6789/0 conn(0x7f5aec066b90 legacy=0x7f5aec0af580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:19.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.143+0000 7f5af3dcf640 1 -- 172.21.15.121:0/135162364 shutdown_connections 2024-03-29T11:48:19.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.143+0000 7f5af3dcf640 1 -- 172.21.15.121:0/135162364 >> 172.21.15.121:0/135162364 conn(0x7f5aec0b4970 msgr2=0x7f5aec0b4d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:19.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.143+0000 7f5af3dcf640 1 -- 172.21.15.121:0/135162364 shutdown_connections 2024-03-29T11:48:19.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.143+0000 7f5af3dcf640 1 -- 172.21.15.121:0/135162364 wait complete. 2024-03-29T11:48:19.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.144+0000 7f5af3dcf640 10 monclient: build_initial_monmap 2024-03-29T11:48:19.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.144+0000 7f5af3dcf640 10 monclient: monmap: 2024-03-29T11:48:19.164 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-29T11:48:19.164 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:48:19.164 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-03-29T11:48:19.164 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-03-29T11:48:19.164 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:19.164 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:19.165 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:48:19.165 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:48:19.165 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:48:19.165 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:19.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.144+0000 7f5af3dcf640 1 librados: starting msgr at 2024-03-29T11:48:19.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.144+0000 7f5af3dcf640 1 librados: starting objecter 2024-03-29T11:48:19.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.144+0000 7f5af3dcf640 1 Processor -- start 2024-03-29T11:48:19.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.144+0000 7f5af3dcf640 1 -- start start 2024-03-29T11:48:19.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.144+0000 7f5af3dcf640 1 librados: setting wanted keys 2024-03-29T11:48:19.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.144+0000 7f5af3dcf640 1 librados: calling monclient init 2024-03-29T11:48:19.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.144+0000 7f5af3dcf640 10 monclient: init 2024-03-29T11:48:19.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.145+0000 7f5af3dcf640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:19.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.145+0000 7f5af3dcf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:48:19.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.145+0000 7f5af3dcf640 10 monclient(hunting): picked mon.noname-c con 0x7f5aec066b90 addr v1:172.21.15.98:6790/0 2024-03-29T11:48:19.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.145+0000 7f5af3dcf640 10 monclient(hunting): picked mon.noname-b con 0x7f5aec13f340 addr v1:172.21.15.121:6789/0 2024-03-29T11:48:19.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.145+0000 7f5af3dcf640 10 monclient(hunting): picked mon.noname-a con 0x7f5aec13f7a0 addr v1:172.21.15.98:6789/0 2024-03-29T11:48:19.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.145+0000 7f5af3dcf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aec133630 con 0x7f5aec13f7a0 2024-03-29T11:48:19.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.145+0000 7f5af3dcf640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aec1346b0 con 0x7f5aec13f340 2024-03-29T11:48:19.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.145+0000 7f5af3dcf640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aec12e7d0 con 0x7f5aec066b90 2024-03-29T11:48:19.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.145+0000 7f5af3dcf640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:19.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.145+0000 7f5af3dcf640 10 monclient(hunting): authenticate will time out at 809.315715s 2024-03-29T11:48:19.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.145+0000 7f5af136a640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f5aec13f340 0x7f5aec12d9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:51474/0 (socket says 172.21.15.121:51474) 2024-03-29T11:48:19.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.145+0000 7f5af136a640 1 -- 172.21.15.121:0/3862065655 learned_addr learned my addr 172.21.15.121:0/3862065655 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-29T11:48:19.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.145+0000 7f5ae27fc640 10 client.?.objecter ms_handle_connect 0x7f5aec13f340 2024-03-29T11:48:19.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.145+0000 7f5ae27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:19.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.145+0000 7f5ae27fc640 10 client.?.objecter ms_handle_connect 0x7f5aec13f7a0 2024-03-29T11:48:19.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.145+0000 7f5ae27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:19.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.145+0000 7f5ae27fc640 10 client.?.objecter ms_handle_connect 0x7f5aec066b90 2024-03-29T11:48:19.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.145+0000 7f5ae27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:19.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.145+0000 7f5ae27fc640 1 -- 172.21.15.121:0/3862065655 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1090972675 0 0) 0x7f5aec1346b0 con 0x7f5aec13f340 2024-03-29T11:48:19.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.145+0000 7f5ae27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:48:19.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.145+0000 7f5ae27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:19.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.145+0000 7f5ae27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:19.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.145+0000 7f5ae27fc640 10 monclient(hunting): my global_id is 4138 2024-03-29T11:48:19.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.145+0000 7f5ae27fc640 1 -- 172.21.15.121:0/3862065655 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ad0004eb0 con 0x7f5aec13f340 2024-03-29T11:48:19.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.145+0000 7f5ae27fc640 1 -- 172.21.15.121:0/3862065655 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4277317367 0 0) 0x7f5aec12e7d0 con 0x7f5aec066b90 2024-03-29T11:48:19.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.145+0000 7f5ae27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:48:19.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.145+0000 7f5ae27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:19.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.145+0000 7f5ae27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:19.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.145+0000 7f5ae27fc640 10 monclient(hunting): my global_id is 4130 2024-03-29T11:48:19.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.145+0000 7f5ae27fc640 1 -- 172.21.15.121:0/3862065655 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5aec1346b0 con 0x7f5aec066b90 2024-03-29T11:48:19.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.145+0000 7f5ae27fc640 1 -- 172.21.15.121:0/3862065655 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 239897886 0 0) 0x7f5aec133630 con 0x7f5aec13f7a0 2024-03-29T11:48:19.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.145+0000 7f5ae27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:48:19.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.145+0000 7f5ae27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:19.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.145+0000 7f5ae27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:19.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.145+0000 7f5ae27fc640 10 monclient(hunting): my global_id is 4131 2024-03-29T11:48:19.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.145+0000 7f5ae27fc640 1 -- 172.21.15.121:0/3862065655 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5aec12e7d0 con 0x7f5aec13f7a0 2024-03-29T11:48:19.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.145+0000 7f5ae27fc640 1 -- 172.21.15.121:0/3862065655 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1135146485 0 0) 0x7f5ad0004eb0 con 0x7f5aec13f340 2024-03-29T11:48:19.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.145+0000 7f5ae27fc640 1 -- 172.21.15.121:0/3862065655 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5aec133630 con 0x7f5aec13f340 2024-03-29T11:48:19.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5ae27fc640 1 -- 172.21.15.121:0/3862065655 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f5ae8001a10 con 0x7f5aec13f340 2024-03-29T11:48:19.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5ae27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:19.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5ae27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:48:19.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5ae27fc640 10 monclient(hunting): dump: 2024-03-29T11:48:19.168 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:19.168 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:19.168 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.168 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.168 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:19.169 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:19.169 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:19.169 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:19.169 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:19.169 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:19.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5ae27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:48:19.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5ae27fc640 1 -- 172.21.15.121:0/3862065655 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3145653225 0 0) 0x7f5aec1346b0 con 0x7f5aec066b90 2024-03-29T11:48:19.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5ae27fc640 1 -- 172.21.15.121:0/3862065655 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ad0004eb0 con 0x7f5aec066b90 2024-03-29T11:48:19.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5ae27fc640 1 -- 172.21.15.121:0/3862065655 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f5adc001b80 con 0x7f5aec066b90 2024-03-29T11:48:19.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5ae27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:19.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5ae27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:48:19.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5ae27fc640 10 monclient(hunting): dump: 2024-03-29T11:48:19.169 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:19.169 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:19.169 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.170 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.170 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:19.170 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:19.170 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:19.170 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:19.170 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:19.170 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:19.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5ae27fc640 1 -- 172.21.15.121:0/3862065655 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3814243178 0 0) 0x7f5aec12e7d0 con 0x7f5aec13f7a0 2024-03-29T11:48:19.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5ae27fc640 1 -- 172.21.15.121:0/3862065655 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5aec1346b0 con 0x7f5aec13f7a0 2024-03-29T11:48:19.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5ae27fc640 1 -- 172.21.15.121:0/3862065655 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f5ae4001a50 con 0x7f5aec13f7a0 2024-03-29T11:48:19.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5ae27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:19.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5ae27fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:48:19.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5ae27fc640 10 monclient(hunting): dump: 2024-03-29T11:48:19.170 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:19.171 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:19.171 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.171 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.171 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:19.171 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:19.171 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:19.171 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:19.171 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:19.171 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:19.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5ae27fc640 1 -- 172.21.15.121:0/3862065655 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1801584993 0 0) 0x7f5aec133630 con 0x7f5aec13f340 2024-03-29T11:48:19.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5ae27fc640 1 -- 172.21.15.121:0/3862065655 >> v1:172.21.15.98:6790/0 conn(0x7f5aec066b90 legacy=0x7f5aec133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:19.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5ae27fc640 1 -- 172.21.15.121:0/3862065655 >> v1:172.21.15.98:6789/0 conn(0x7f5aec13f7a0 legacy=0x7f5aec12e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:19.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5ae27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:48:19.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5ae27fc640 1 monclient: found mon.b 2024-03-29T11:48:19.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5ae27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:19.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5ae27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:19.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5ae27fc640 1 -- 172.21.15.121:0/3862065655 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aec1320d0 con 0x7f5aec13f340 2024-03-29T11:48:19.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5ae27fc640 10 monclient: _finish_auth 0 2024-03-29T11:48:19.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5ae27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:48:19.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5ae27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:19.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5af3dcf640 5 monclient: authenticate success, global_id 4138 2024-03-29T11:48:19.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5af3dcf640 10 monclient: _renew_subs 2024-03-29T11:48:19.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5af3dcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:19.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5af3dcf640 1 -- 172.21.15.121:0/3862065655 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aec1338a0 con 0x7f5aec13f340 2024-03-29T11:48:19.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5af3dcf640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:48:19.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5af3dcf640 10 monclient: _renew_subs 2024-03-29T11:48:19.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5af3dcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:19.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5af3dcf640 1 -- 172.21.15.121:0/3862065655 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aec1426b0 con 0x7f5aec13f340 2024-03-29T11:48:19.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5af3dcf640 1 librados: init done 2024-03-29T11:48:19.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5ae27fc640 1 -- 172.21.15.121:0/3862065655 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ae8001cf0 con 0x7f5aec13f340 2024-03-29T11:48:19.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5ae27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:19.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5ae27fc640 1 -- 172.21.15.121:0/3862065655 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f5ae8003a60 con 0x7f5aec13f340 2024-03-29T11:48:19.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5ae27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:19.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5ae27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:48:19.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5ae3fff640 4 set_mon_vals no callback set 2024-03-29T11:48:19.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5ae27fc640 10 monclient: dump: 2024-03-29T11:48:19.173 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:19.173 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:19.173 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.173 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.173 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:19.173 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:19.173 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:19.174 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:19.174 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:19.174 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:19.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5ae27fc640 1 -- 172.21.15.121:0/3862065655 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f5ae8004010 con 0x7f5aec13f340 2024-03-29T11:48:19.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5ae27fc640 1 -- 172.21.15.121:0/3862065655 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 1042294928 0 0) 0x7f5ae8006870 con 0x7f5aec13f340 2024-03-29T11:48:19.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5ae27fc640 10 client.4138.objecter ms_dispatch 0x7f5aec000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T11:48:19.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5ae27fc640 3 client.4138.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T11:48:19.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5ae27fc640 3 client.4138.objecter handle_osd_map decoding full epoch 5 2024-03-29T11:48:19.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.146+0000 7f5ae27fc640 20 client.4138.objecter dump_active .. 0 homeless 2024-03-29T11:48:19.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.147+0000 7f5af3dcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:19.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.147+0000 7f5af3dcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:19.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.147+0000 7f5af3dcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:19.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.147+0000 7f5af3dcf640 1 -- 172.21.15.121:0/3862065655 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ad4005080 con 0x7f5aec13f340 2024-03-29T11:48:19.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.149+0000 7f5ae27fc640 1 -- 172.21.15.121:0/3862065655 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f5ae80063e0 con 0x7f5aec13f340 2024-03-29T11:48:19.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.149+0000 7f5ae27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:19.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.149+0000 7f5ae27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:48:19.175 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:48:19.152+0000 7f4e33c10040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:48:19.201 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:48:19.195+0000 7f4e33c10040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:48:19.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.220+0000 7f5af3dcf640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c6c10299-76b6-4de5-ac71-6de9c36b6de4", "id": 3}] 2024-03-29T11:48:19.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.220+0000 7f5af3dcf640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c6c10299-76b6-4de5-ac71-6de9c36b6de4", "id": 3}] 2024-03-29T11:48:19.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.220+0000 7f5af3dcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:19.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.220+0000 7f5af3dcf640 1 -- 172.21.15.121:0/3862065655 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd new", "uuid": "c6c10299-76b6-4de5-ac71-6de9c36b6de4", "id": 3} v 0) v1 -- 0x7f5ad40058b0 con 0x7f5aec13f340 2024-03-29T11:48:19.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.224+0000 7f5ae27fc640 1 -- 172.21.15.121:0/3862065655 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c6c10299-76b6-4de5-ac71-6de9c36b6de4", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 3186752203 0 525046306) 0x7f5ae80066c0 con 0x7f5aec13f340 2024-03-29T11:48:19.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.224+0000 7f5ae27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c6c10299-76b6-4de5-ac71-6de9c36b6de4", "id": 3}] 2024-03-29T11:48:19.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.224+0000 7f5ae27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:48:19.230 INFO:teuthology.orchestra.run.smithi121.stdout:3 2024-03-29T11:48:19.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.225+0000 7f5af3dcf640 10 librados: watch_flush enter 2024-03-29T11:48:19.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.225+0000 7f5af3dcf640 10 librados: watch_flush exit 2024-03-29T11:48:19.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.225+0000 7f5af3dcf640 20 client.4138.objecter shutdown clearing up homeless session... 2024-03-29T11:48:19.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.225+0000 7f5af3dcf640 10 client.4138.objecter successfully canceled tick 2024-03-29T11:48:19.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.225+0000 7f5af3dcf640 10 monclient: shutdown 2024-03-29T11:48:19.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.225+0000 7f5af3dcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:19.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.225+0000 7f5af3dcf640 1 -- 172.21.15.121:0/3862065655 >> v1:172.21.15.121:6789/0 conn(0x7f5aec13f340 legacy=0x7f5aec12d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:19.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.225+0000 7f5af3dcf640 1 -- 172.21.15.121:0/3862065655 shutdown_connections 2024-03-29T11:48:19.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.225+0000 7f5af3dcf640 1 -- 172.21.15.121:0/3862065655 >> 172.21.15.121:0/3862065655 conn(0x7f5aec0b4970 msgr2=0x7f5aec0af270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:19.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.225+0000 7f5af3dcf640 1 -- 172.21.15.121:0/3862065655 shutdown_connections 2024-03-29T11:48:19.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.225+0000 7f5af3dcf640 1 -- 172.21.15.121:0/3862065655 wait complete. 2024-03-29T11:48:19.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.225+0000 7f5af3dcf640 1 librados: shutdown 2024-03-29T11:48:19.241 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 0156b67c-8892-45ea-8bff-c2e991ee33a3 4 2024-03-29T11:48:19.247 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T11:48:19.242+0000 7faaea042040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:48:19.317 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T11:48:19.311+0000 7faaea042040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:48:19.321 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:48:19.316+0000 7f4e33c10040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:48:19.362 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T11:48:19.356+0000 7faaea042040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:48:19.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.376+0000 7fe2001cf640 10 monclient: get_monmap_and_config 2024-03-29T11:48:19.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.376+0000 7fe2001cf640 10 monclient: build_initial_monmap 2024-03-29T11:48:19.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.376+0000 7fe2001cf640 10 monclient: monmap: 2024-03-29T11:48:19.382 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-29T11:48:19.382 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:19.382 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:48:19.377176+0000 2024-03-29T11:48:19.382 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:48:19.377176+0000 2024-03-29T11:48:19.382 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:19.382 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:19.382 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:48:19.382 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:48:19.382 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:48:19.382 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:19.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.376+0000 7fe2001cf640 1 Processor -- start 2024-03-29T11:48:19.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.376+0000 7fe2001cf640 1 -- start start 2024-03-29T11:48:19.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.376+0000 7fe2001cf640 10 monclient: init 2024-03-29T11:48:19.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.377+0000 7fe2001cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:19.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.377+0000 7fe2001cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:48:19.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.377+0000 7fe2001cf640 10 monclient(hunting): picked mon.noname-c con 0x7fe1f809ac40 addr v1:172.21.15.121:6789/0 2024-03-29T11:48:19.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.377+0000 7fe2001cf640 10 monclient(hunting): picked mon.noname-a con 0x7fe1f809c870 addr v1:172.21.15.98:6789/0 2024-03-29T11:48:19.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.377+0000 7fe2001cf640 10 monclient(hunting): picked mon.noname-b con 0x7fe1f80b4360 addr v1:172.21.15.98:6790/0 2024-03-29T11:48:19.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.377+0000 7fe2001cf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f809dec0 con 0x7fe1f809c870 2024-03-29T11:48:19.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.377+0000 7fe2001cf640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f809da30 con 0x7fe1f809ac40 2024-03-29T11:48:19.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.377+0000 7fe2001cf640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f80b39c0 con 0x7fe1f80b4360 2024-03-29T11:48:19.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.377+0000 7fe2001cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:19.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.377+0000 7fe2001cf640 10 monclient(hunting): authenticate will time out at 809.547767s 2024-03-29T11:48:19.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.377+0000 7fe1fdf6b640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fe1f809ac40 0x7fe1f80b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:51486/0 (socket says 172.21.15.121:51486) 2024-03-29T11:48:19.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.377+0000 7fe1fdf6b640 1 -- 172.21.15.121:0/2942418011 learned_addr learned my addr 172.21.15.121:0/2942418011 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-29T11:48:19.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.377+0000 7fe1fcf69640 1 -- 172.21.15.121:0/2942418011 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4058194913 0 0) 0x7fe1f809da30 con 0x7fe1f809ac40 2024-03-29T11:48:19.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.377+0000 7fe1fcf69640 10 monclient(hunting): _negotiate 2024-03-29T11:48:19.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.377+0000 7fe1fcf69640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:19.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.377+0000 7fe1fcf69640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:19.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.377+0000 7fe1fcf69640 10 monclient(hunting): my global_id is 4141 2024-03-29T11:48:19.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.377+0000 7fe1fcf69640 1 -- 172.21.15.121:0/2942418011 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1e4003520 con 0x7fe1f809ac40 2024-03-29T11:48:19.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.377+0000 7fe1fcf69640 1 -- 172.21.15.121:0/2942418011 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2926527816 0 0) 0x7fe1f80b39c0 con 0x7fe1f80b4360 2024-03-29T11:48:19.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.377+0000 7fe1fcf69640 10 monclient(hunting): _negotiate 2024-03-29T11:48:19.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.377+0000 7fe1fcf69640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:19.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.377+0000 7fe1fcf69640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:19.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.377+0000 7fe1fcf69640 10 monclient(hunting): my global_id is 4133 2024-03-29T11:48:19.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.377+0000 7fe1fcf69640 1 -- 172.21.15.121:0/2942418011 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1f809da30 con 0x7fe1f80b4360 2024-03-29T11:48:19.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.377+0000 7fe1fcf69640 1 -- 172.21.15.121:0/2942418011 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 322720364 0 0) 0x7fe1f4001110 con 0x7fe1f809c870 2024-03-29T11:48:19.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.377+0000 7fe1fcf69640 10 monclient(hunting): _negotiate 2024-03-29T11:48:19.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.377+0000 7fe1fcf69640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:19.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.377+0000 7fe1fcf69640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:19.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.377+0000 7fe1fcf69640 10 monclient(hunting): my global_id is 4134 2024-03-29T11:48:19.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.377+0000 7fe1fcf69640 1 -- 172.21.15.121:0/2942418011 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1f80b39c0 con 0x7fe1f809c870 2024-03-29T11:48:19.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.378+0000 7fe1fcf69640 1 -- 172.21.15.121:0/2942418011 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4095098542 0 0) 0x7fe1e4003520 con 0x7fe1f809ac40 2024-03-29T11:48:19.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.378+0000 7fe1fcf69640 1 -- 172.21.15.121:0/2942418011 >> v1:172.21.15.98:6790/0 conn(0x7fe1f80b4360 legacy=0x7fe1f814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:19.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.378+0000 7fe1fcf69640 1 -- 172.21.15.121:0/2942418011 >> v1:172.21.15.98:6789/0 conn(0x7fe1f809c870 legacy=0x7fe1f813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:19.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.378+0000 7fe1fcf69640 10 monclient: _finish_hunting 0 2024-03-29T11:48:19.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.378+0000 7fe1fcf69640 1 monclient: found mon.noname-c 2024-03-29T11:48:19.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.378+0000 7fe1fcf69640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:19.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.378+0000 7fe1fcf69640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:48:19.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.378+0000 7fe1fcf69640 1 -- 172.21.15.121:0/2942418011 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f80b5160 con 0x7fe1f809ac40 2024-03-29T11:48:19.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.378+0000 7fe1fcf69640 10 monclient: _finish_auth 0 2024-03-29T11:48:19.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.378+0000 7fe1fcf69640 10 monclient: _check_auth_tickets 2024-03-29T11:48:19.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.378+0000 7fe1fcf69640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:19.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.378+0000 7fe2001cf640 5 monclient: authenticate success, global_id 4141 2024-03-29T11:48:19.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.378+0000 7fe1fcf69640 1 -- 172.21.15.121:0/2942418011 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fe1ec001690 con 0x7fe1f809ac40 2024-03-29T11:48:19.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.378+0000 7fe2001cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:48:19.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.378+0000 7fe1fcf69640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:19.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.378+0000 7fe1fcf69640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:48:19.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.378+0000 7fe1fcf69640 10 monclient: dump: 2024-03-29T11:48:19.398 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:19.398 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:19.398 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.398 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.398 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:19.398 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:19.399 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:19.399 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:19.399 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:19.399 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:19.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.378+0000 7fe2001cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:48:19.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.378+0000 7fe1fcf69640 1 -- 172.21.15.121:0/2942418011 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1ec0017f0 con 0x7fe1f809ac40 2024-03-29T11:48:19.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.378+0000 7fe1fcf69640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:19.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.378+0000 7fe1fcf69640 1 -- 172.21.15.121:0/2942418011 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fe1ec001c00 con 0x7fe1f809ac40 2024-03-29T11:48:19.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.378+0000 7fe2001cf640 10 monclient: get_monmap_and_config success 2024-03-29T11:48:19.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.378+0000 7fe2001cf640 4 set_mon_vals no callback set 2024-03-29T11:48:19.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.378+0000 7fe1fcf69640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:19.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.378+0000 7fe1fcf69640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:48:19.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.378+0000 7fe1fcf69640 10 monclient: dump: 2024-03-29T11:48:19.399 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:19.400 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:19.400 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.400 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.400 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:19.400 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:19.400 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:19.400 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:19.400 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:19.400 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:19.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.378+0000 7fe2001cf640 10 monclient: shutdown 2024-03-29T11:48:19.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.378+0000 7fe2001cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:19.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.378+0000 7fe2001cf640 1 -- 172.21.15.121:0/2942418011 >> v1:172.21.15.121:6789/0 conn(0x7fe1f809ac40 legacy=0x7fe1f80b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:19.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.378+0000 7fe2001cf640 1 -- 172.21.15.121:0/2942418011 shutdown_connections 2024-03-29T11:48:19.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.378+0000 7fe2001cf640 1 -- 172.21.15.121:0/2942418011 >> 172.21.15.121:0/2942418011 conn(0x7fe1f8066b00 msgr2=0x7fe1f8066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:19.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.378+0000 7fe2001cf640 1 -- 172.21.15.121:0/2942418011 shutdown_connections 2024-03-29T11:48:19.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.378+0000 7fe2001cf640 1 -- 172.21.15.121:0/2942418011 wait complete. 2024-03-29T11:48:19.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.379+0000 7fe2001cf640 10 monclient: build_initial_monmap 2024-03-29T11:48:19.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.379+0000 7fe2001cf640 10 monclient: monmap: 2024-03-29T11:48:19.401 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-29T11:48:19.401 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:48:19.401 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-03-29T11:48:19.401 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-03-29T11:48:19.401 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:19.401 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:19.401 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:48:19.401 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:48:19.401 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:48:19.401 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:19.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.379+0000 7fe2001cf640 1 librados: starting msgr at 2024-03-29T11:48:19.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.379+0000 7fe2001cf640 1 librados: starting objecter 2024-03-29T11:48:19.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.379+0000 7fe2001cf640 1 Processor -- start 2024-03-29T11:48:19.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.379+0000 7fe2001cf640 1 -- start start 2024-03-29T11:48:19.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.379+0000 7fe2001cf640 1 librados: setting wanted keys 2024-03-29T11:48:19.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.379+0000 7fe2001cf640 1 librados: calling monclient init 2024-03-29T11:48:19.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.379+0000 7fe2001cf640 10 monclient: init 2024-03-29T11:48:19.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.380+0000 7fe2001cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:19.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.380+0000 7fe2001cf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:48:19.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.380+0000 7fe2001cf640 10 monclient(hunting): picked mon.noname-a con 0x7fe1f809ac40 addr v1:172.21.15.98:6789/0 2024-03-29T11:48:19.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.380+0000 7fe2001cf640 10 monclient(hunting): picked mon.noname-c con 0x7fe1f80b8540 addr v1:172.21.15.98:6790/0 2024-03-29T11:48:19.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.380+0000 7fe2001cf640 10 monclient(hunting): picked mon.noname-b con 0x7fe1f80b5ab0 addr v1:172.21.15.121:6789/0 2024-03-29T11:48:19.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.380+0000 7fe2001cf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f80b76f0 con 0x7fe1f809ac40 2024-03-29T11:48:19.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.380+0000 7fe2001cf640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f80b80b0 con 0x7fe1f80b5ab0 2024-03-29T11:48:19.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.380+0000 7fe2001cf640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f80b7940 con 0x7fe1f80b8540 2024-03-29T11:48:19.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.380+0000 7fe2001cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:19.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.380+0000 7fe2001cf640 10 monclient(hunting): authenticate will time out at 809.550721s 2024-03-29T11:48:19.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.380+0000 7fe1fe76c640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fe1f80b5ab0 0x7fe1f81e2b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:51494/0 (socket says 172.21.15.121:51494) 2024-03-29T11:48:19.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.380+0000 7fe1fe76c640 1 -- 172.21.15.121:0/1186690535 learned_addr learned my addr 172.21.15.121:0/1186690535 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-29T11:48:19.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.380+0000 7fe1ea7fc640 10 client.?.objecter ms_handle_connect 0x7fe1f80b5ab0 2024-03-29T11:48:19.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.380+0000 7fe1ea7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:19.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.380+0000 7fe1ea7fc640 10 client.?.objecter ms_handle_connect 0x7fe1f809ac40 2024-03-29T11:48:19.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.380+0000 7fe1ea7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:19.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.380+0000 7fe1ea7fc640 10 client.?.objecter ms_handle_connect 0x7fe1f80b8540 2024-03-29T11:48:19.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.380+0000 7fe1ea7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:19.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.380+0000 7fe1ea7fc640 1 -- 172.21.15.121:0/1186690535 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1285147813 0 0) 0x7fe1f80b80b0 con 0x7fe1f80b5ab0 2024-03-29T11:48:19.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.380+0000 7fe1ea7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:48:19.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.380+0000 7fe1ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:19.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.380+0000 7fe1ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:19.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.380+0000 7fe1ea7fc640 10 monclient(hunting): my global_id is 4144 2024-03-29T11:48:19.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.380+0000 7fe1ea7fc640 1 -- 172.21.15.121:0/1186690535 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1dc004eb0 con 0x7fe1f80b5ab0 2024-03-29T11:48:19.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.380+0000 7fe1ea7fc640 1 -- 172.21.15.121:0/1186690535 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3604942030 0 0) 0x7fe1f80b76f0 con 0x7fe1f809ac40 2024-03-29T11:48:19.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.380+0000 7fe1ea7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:48:19.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.380+0000 7fe1ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:19.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.380+0000 7fe1ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:19.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.380+0000 7fe1ea7fc640 10 monclient(hunting): my global_id is 4137 2024-03-29T11:48:19.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.380+0000 7fe1ea7fc640 1 -- 172.21.15.121:0/1186690535 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1f80b80b0 con 0x7fe1f809ac40 2024-03-29T11:48:19.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.380+0000 7fe1ea7fc640 1 -- 172.21.15.121:0/1186690535 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1414454737 0 0) 0x7fe1f80b7940 con 0x7fe1f80b8540 2024-03-29T11:48:19.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.380+0000 7fe1ea7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:48:19.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.380+0000 7fe1ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:19.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.380+0000 7fe1ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:19.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.380+0000 7fe1ea7fc640 10 monclient(hunting): my global_id is 4136 2024-03-29T11:48:19.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.380+0000 7fe1ea7fc640 1 -- 172.21.15.121:0/1186690535 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1f80b76f0 con 0x7fe1f80b8540 2024-03-29T11:48:19.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.380+0000 7fe1ea7fc640 1 -- 172.21.15.121:0/1186690535 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1225437487 0 0) 0x7fe1dc004eb0 con 0x7fe1f80b5ab0 2024-03-29T11:48:19.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.380+0000 7fe1ea7fc640 1 -- 172.21.15.121:0/1186690535 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1f80b7940 con 0x7fe1f80b5ab0 2024-03-29T11:48:19.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.380+0000 7fe1ea7fc640 1 -- 172.21.15.121:0/1186690535 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fe1f0001c90 con 0x7fe1f80b5ab0 2024-03-29T11:48:19.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.380+0000 7fe1ea7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:19.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe1ea7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:48:19.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe1ea7fc640 10 monclient(hunting): dump: 2024-03-29T11:48:19.405 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:19.405 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:19.405 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.405 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.405 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:19.405 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:19.405 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:19.405 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:19.405 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:19.405 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:19.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe1ea7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:48:19.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe1ea7fc640 1 -- 172.21.15.121:0/1186690535 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 878526767 0 0) 0x7fe1f80b76f0 con 0x7fe1f80b8540 2024-03-29T11:48:19.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe1ea7fc640 1 -- 172.21.15.121:0/1186690535 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1dc004eb0 con 0x7fe1f80b8540 2024-03-29T11:48:19.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe1ea7fc640 1 -- 172.21.15.121:0/1186690535 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fe1f4001860 con 0x7fe1f80b8540 2024-03-29T11:48:19.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe1ea7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:19.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe1ea7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:48:19.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe1ea7fc640 10 monclient(hunting): dump: 2024-03-29T11:48:19.406 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:19.406 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:19.406 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.406 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.406 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:19.406 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:19.406 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:19.406 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:19.406 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:19.407 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:19.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe1ea7fc640 1 -- 172.21.15.121:0/1186690535 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3062943203 0 0) 0x7fe1f80b80b0 con 0x7fe1f809ac40 2024-03-29T11:48:19.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe1ea7fc640 1 -- 172.21.15.121:0/1186690535 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1f80b76f0 con 0x7fe1f809ac40 2024-03-29T11:48:19.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe1ea7fc640 1 -- 172.21.15.121:0/1186690535 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fe1ec001b00 con 0x7fe1f809ac40 2024-03-29T11:48:19.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe1ea7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:19.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe1ea7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:48:19.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe1ea7fc640 10 monclient(hunting): dump: 2024-03-29T11:48:19.407 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:19.407 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:19.407 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.407 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.407 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:19.407 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:19.407 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:19.408 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:19.408 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:19.408 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:19.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe1ea7fc640 1 -- 172.21.15.121:0/1186690535 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3975233400 0 0) 0x7fe1f80b7940 con 0x7fe1f80b5ab0 2024-03-29T11:48:19.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe1ea7fc640 1 -- 172.21.15.121:0/1186690535 >> v1:172.21.15.98:6790/0 conn(0x7fe1f80b8540 legacy=0x7fe1f8139d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:19.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe1ea7fc640 1 -- 172.21.15.121:0/1186690535 >> v1:172.21.15.98:6789/0 conn(0x7fe1f809ac40 legacy=0x7fe1f81d7300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:19.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe1ea7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:48:19.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe1ea7fc640 1 monclient: found mon.b 2024-03-29T11:48:19.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe1ea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:19.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe1ea7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:19.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe1ea7fc640 1 -- 172.21.15.121:0/1186690535 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f80b7c20 con 0x7fe1f80b5ab0 2024-03-29T11:48:19.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe1ea7fc640 10 monclient: _finish_auth 0 2024-03-29T11:48:19.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe1ea7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:48:19.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe1ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:19.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe2001cf640 5 monclient: authenticate success, global_id 4144 2024-03-29T11:48:19.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe2001cf640 10 monclient: _renew_subs 2024-03-29T11:48:19.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe2001cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:19.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe2001cf640 1 -- 172.21.15.121:0/1186690535 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1f81d7a10 con 0x7fe1f80b5ab0 2024-03-29T11:48:19.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe2001cf640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:48:19.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe2001cf640 10 monclient: _renew_subs 2024-03-29T11:48:19.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe2001cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:19.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe2001cf640 1 -- 172.21.15.121:0/1186690535 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1f80b60c0 con 0x7fe1f80b5ab0 2024-03-29T11:48:19.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe2001cf640 1 librados: init done 2024-03-29T11:48:19.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe1ea7fc640 1 -- 172.21.15.121:0/1186690535 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1f0003970 con 0x7fe1f80b5ab0 2024-03-29T11:48:19.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe1ea7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:19.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe1ea7fc640 1 -- 172.21.15.121:0/1186690535 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fe1f0003cc0 con 0x7fe1f80b5ab0 2024-03-29T11:48:19.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe1ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:19.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe1ea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:48:19.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe1ea7fc640 10 monclient: dump: 2024-03-29T11:48:19.409 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:19.410 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:19.410 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.410 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.410 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:19.410 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:19.410 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:19.410 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:19.410 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:19.410 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:19.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe1ebfff640 4 set_mon_vals no callback set 2024-03-29T11:48:19.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe1ea7fc640 1 -- 172.21.15.121:0/1186690535 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe1f0004230 con 0x7fe1f80b5ab0 2024-03-29T11:48:19.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe1ea7fc640 1 -- 172.21.15.121:0/1186690535 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 606608434 0 0) 0x7fe1f0006af0 con 0x7fe1f80b5ab0 2024-03-29T11:48:19.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe1ea7fc640 10 client.4144.objecter ms_dispatch 0x7fe1f8000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T11:48:19.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe1ea7fc640 3 client.4144.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T11:48:19.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe1ea7fc640 3 client.4144.objecter handle_osd_map decoding full epoch 6 2024-03-29T11:48:19.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.381+0000 7fe1ea7fc640 20 client.4144.objecter dump_active .. 0 homeless 2024-03-29T11:48:19.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.382+0000 7fe2001cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:19.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.382+0000 7fe2001cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:19.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.382+0000 7fe2001cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:19.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.382+0000 7fe2001cf640 1 -- 172.21.15.121:0/1186690535 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1f809b020 con 0x7fe1f80b5ab0 2024-03-29T11:48:19.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.384+0000 7fe1ea7fc640 1 -- 172.21.15.121:0/1186690535 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe1f0004710 con 0x7fe1f80b5ab0 2024-03-29T11:48:19.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.384+0000 7fe1ea7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:19.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.384+0000 7fe1ea7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:48:19.411 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:48:19.390+0000 7f4e33c10040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:48:19.414 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T11:48:19.409+0000 7faaea042040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:48:19.443 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:48:19.438+0000 7f4e33c10040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:48:19.461 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T11:48:19.455+0000 7faaea042040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:48:19.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.456+0000 7fe2001cf640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "0156b67c-8892-45ea-8bff-c2e991ee33a3", "id": 4}] 2024-03-29T11:48:19.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.456+0000 7fe2001cf640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "0156b67c-8892-45ea-8bff-c2e991ee33a3", "id": 4}] 2024-03-29T11:48:19.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.456+0000 7fe2001cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:19.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.456+0000 7fe2001cf640 1 -- 172.21.15.121:0/1186690535 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd new", "uuid": "0156b67c-8892-45ea-8bff-c2e991ee33a3", "id": 4} v 0) v1 -- 0x7fe1f80b63c0 con 0x7fe1f80b5ab0 2024-03-29T11:48:19.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.460+0000 7fe1ea7fc640 1 -- 172.21.15.121:0/1186690535 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0156b67c-8892-45ea-8bff-c2e991ee33a3", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 3526448365 0 1697012583) 0x7fe1f0006710 con 0x7fe1f80b5ab0 2024-03-29T11:48:19.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.460+0000 7fe1ea7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "0156b67c-8892-45ea-8bff-c2e991ee33a3", "id": 4}] 2024-03-29T11:48:19.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.460+0000 7fe1ea7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:48:19.466 INFO:teuthology.orchestra.run.smithi121.stdout:4 2024-03-29T11:48:19.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.460+0000 7fe2001cf640 10 librados: watch_flush enter 2024-03-29T11:48:19.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.460+0000 7fe2001cf640 10 librados: watch_flush exit 2024-03-29T11:48:19.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.460+0000 7fe2001cf640 20 client.4144.objecter shutdown clearing up homeless session... 2024-03-29T11:48:19.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.460+0000 7fe2001cf640 10 client.4144.objecter successfully canceled tick 2024-03-29T11:48:19.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.460+0000 7fe2001cf640 10 monclient: shutdown 2024-03-29T11:48:19.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.460+0000 7fe2001cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:19.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.460+0000 7fe2001cf640 1 -- 172.21.15.121:0/1186690535 >> v1:172.21.15.121:6789/0 conn(0x7fe1f80b5ab0 legacy=0x7fe1f81e2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:19.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.461+0000 7fe2001cf640 1 -- 172.21.15.121:0/1186690535 shutdown_connections 2024-03-29T11:48:19.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.461+0000 7fe2001cf640 1 -- 172.21.15.121:0/1186690535 >> 172.21.15.121:0/1186690535 conn(0x7fe1f8066b00 msgr2=0x7fe1f813c380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:19.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.461+0000 7fe2001cf640 1 -- 172.21.15.121:0/1186690535 shutdown_connections 2024-03-29T11:48:19.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.461+0000 7fe2001cf640 1 -- 172.21.15.121:0/1186690535 wait complete. 2024-03-29T11:48:19.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.461+0000 7fe2001cf640 1 librados: shutdown 2024-03-29T11:48:19.476 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 510d2deb-540c-43f6-8d32-6783917c2208 5 2024-03-29T11:48:19.498 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:48:19.492+0000 7f4e33c10040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:48:19.548 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:48:19.542+0000 7f4e33c10040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:48:19.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.592+0000 7f058dc06640 10 monclient: get_monmap_and_config 2024-03-29T11:48:19.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.592+0000 7f058dc06640 10 monclient: build_initial_monmap 2024-03-29T11:48:19.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.592+0000 7f058dc06640 10 monclient: monmap: 2024-03-29T11:48:19.598 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-29T11:48:19.598 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:19.598 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:48:19.593779+0000 2024-03-29T11:48:19.598 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:48:19.593779+0000 2024-03-29T11:48:19.599 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:19.599 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:19.599 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:48:19.599 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:48:19.599 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:48:19.599 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:19.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.593+0000 7f058dc06640 1 Processor -- start 2024-03-29T11:48:19.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.593+0000 7f058dc06640 1 -- start start 2024-03-29T11:48:19.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.593+0000 7f058dc06640 10 monclient: init 2024-03-29T11:48:19.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f058dc06640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:19.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f058dc06640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:48:19.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f058dc06640 10 monclient(hunting): picked mon.noname-b con 0x7f058809ac40 addr v1:172.21.15.98:6790/0 2024-03-29T11:48:19.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f058dc06640 10 monclient(hunting): picked mon.noname-c con 0x7f058809c870 addr v1:172.21.15.121:6789/0 2024-03-29T11:48:19.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f058dc06640 10 monclient(hunting): picked mon.noname-a con 0x7f05880b4360 addr v1:172.21.15.98:6789/0 2024-03-29T11:48:19.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f058dc06640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f058809dec0 con 0x7f05880b4360 2024-03-29T11:48:19.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f058dc06640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f058809da30 con 0x7f058809c870 2024-03-29T11:48:19.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f058dc06640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05880b39c0 con 0x7f058809ac40 2024-03-29T11:48:19.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f058dc06640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:19.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f058dc06640 10 monclient(hunting): authenticate will time out at 809.764407s 2024-03-29T11:48:19.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f057effd640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f058809c870 0x7f058813f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:51498/0 (socket says 172.21.15.121:51498) 2024-03-29T11:48:19.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f057effd640 1 -- 172.21.15.121:0/1023296254 learned_addr learned my addr 172.21.15.121:0/1023296254 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-29T11:48:19.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f0586ffd640 1 -- 172.21.15.121:0/1023296254 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3583372385 0 0) 0x7f058809da30 con 0x7f058809c870 2024-03-29T11:48:19.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f0586ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:48:19.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f0586ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:19.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f0586ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:19.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f0586ffd640 10 monclient(hunting): my global_id is 4147 2024-03-29T11:48:19.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f0586ffd640 1 -- 172.21.15.121:0/1023296254 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0568003520 con 0x7f058809c870 2024-03-29T11:48:19.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f0586ffd640 1 -- 172.21.15.121:0/1023296254 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2939288512 0 0) 0x7f05880b39c0 con 0x7f058809ac40 2024-03-29T11:48:19.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f0586ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:48:19.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f0586ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:19.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f0586ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:19.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f0586ffd640 10 monclient(hunting): my global_id is 4139 2024-03-29T11:48:19.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f0586ffd640 1 -- 172.21.15.121:0/1023296254 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f058809da30 con 0x7f058809ac40 2024-03-29T11:48:19.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f0586ffd640 1 -- 172.21.15.121:0/1023296254 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1036293320 0 0) 0x7f0580001490 con 0x7f05880b4360 2024-03-29T11:48:19.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f0586ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:48:19.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f0586ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:19.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f0586ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:19.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f0586ffd640 10 monclient(hunting): my global_id is 4140 2024-03-29T11:48:19.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f0586ffd640 1 -- 172.21.15.121:0/1023296254 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05880b39c0 con 0x7f05880b4360 2024-03-29T11:48:19.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f0586ffd640 1 -- 172.21.15.121:0/1023296254 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1952686809 0 0) 0x7f0568003520 con 0x7f058809c870 2024-03-29T11:48:19.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f0586ffd640 1 -- 172.21.15.121:0/1023296254 >> v1:172.21.15.98:6790/0 conn(0x7f058809ac40 legacy=0x7f05880b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:19.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f0586ffd640 1 -- 172.21.15.121:0/1023296254 >> v1:172.21.15.98:6789/0 conn(0x7f05880b4360 legacy=0x7f058814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:19.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f0586ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:48:19.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f0586ffd640 1 monclient: found mon.noname-c 2024-03-29T11:48:19.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f0586ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:19.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f0586ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:48:19.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f0586ffd640 1 -- 172.21.15.121:0/1023296254 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05880b5160 con 0x7f058809c870 2024-03-29T11:48:19.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f0586ffd640 10 monclient: _finish_auth 0 2024-03-29T11:48:19.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f0586ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:48:19.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f0586ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:19.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f0586ffd640 1 -- 172.21.15.121:0/1023296254 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f0570001690 con 0x7f058809c870 2024-03-29T11:48:19.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f058dc06640 5 monclient: authenticate success, global_id 4147 2024-03-29T11:48:19.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f058dc06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:48:19.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f0586ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:19.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f0586ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:48:19.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f0586ffd640 10 monclient: dump: 2024-03-29T11:48:19.610 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:19.610 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:19.610 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.611 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.611 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:19.611 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:19.611 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:19.611 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:19.611 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:19.611 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:19.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f058dc06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:48:19.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f0586ffd640 1 -- 172.21.15.121:0/1023296254 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0570001870 con 0x7f058809c870 2024-03-29T11:48:19.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f0586ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:19.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f058dc06640 10 monclient: get_monmap_and_config success 2024-03-29T11:48:19.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f0586ffd640 1 -- 172.21.15.121:0/1023296254 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f0570001c80 con 0x7f058809c870 2024-03-29T11:48:19.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f058dc06640 4 set_mon_vals no callback set 2024-03-29T11:48:19.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f0586ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:19.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f0586ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:48:19.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.594+0000 7f0586ffd640 10 monclient: dump: 2024-03-29T11:48:19.612 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:19.612 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:19.612 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.612 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.612 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:19.612 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:19.612 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:19.612 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:19.612 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:19.612 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:19.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.595+0000 7f058dc06640 10 monclient: shutdown 2024-03-29T11:48:19.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.595+0000 7f058dc06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:19.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.595+0000 7f058dc06640 1 -- 172.21.15.121:0/1023296254 >> v1:172.21.15.121:6789/0 conn(0x7f058809c870 legacy=0x7f058813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:19.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.595+0000 7f058dc06640 1 -- 172.21.15.121:0/1023296254 shutdown_connections 2024-03-29T11:48:19.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.595+0000 7f058dc06640 1 -- 172.21.15.121:0/1023296254 >> 172.21.15.121:0/1023296254 conn(0x7f0588066b00 msgr2=0x7f0588066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:19.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.595+0000 7f058dc06640 1 -- 172.21.15.121:0/1023296254 shutdown_connections 2024-03-29T11:48:19.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.595+0000 7f058dc06640 1 -- 172.21.15.121:0/1023296254 wait complete. 2024-03-29T11:48:19.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.595+0000 7f058dc06640 10 monclient: build_initial_monmap 2024-03-29T11:48:19.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.595+0000 7f058dc06640 10 monclient: monmap: 2024-03-29T11:48:19.613 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-29T11:48:19.613 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:48:19.613 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-03-29T11:48:19.613 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-03-29T11:48:19.613 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:19.613 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:19.613 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:48:19.613 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:48:19.613 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:48:19.614 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:19.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.596+0000 7f058dc06640 1 librados: starting msgr at 2024-03-29T11:48:19.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.596+0000 7f058dc06640 1 librados: starting objecter 2024-03-29T11:48:19.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.596+0000 7f058dc06640 1 Processor -- start 2024-03-29T11:48:19.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.596+0000 7f058dc06640 1 -- start start 2024-03-29T11:48:19.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.596+0000 7f058dc06640 1 librados: setting wanted keys 2024-03-29T11:48:19.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.596+0000 7f058dc06640 1 librados: calling monclient init 2024-03-29T11:48:19.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.596+0000 7f058dc06640 10 monclient: init 2024-03-29T11:48:19.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f058dc06640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:19.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f058dc06640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:48:19.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f058dc06640 10 monclient(hunting): picked mon.noname-b con 0x7f058809ac40 addr v1:172.21.15.121:6789/0 2024-03-29T11:48:19.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f058dc06640 10 monclient(hunting): picked mon.noname-a con 0x7f0588140660 addr v1:172.21.15.98:6789/0 2024-03-29T11:48:19.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f058dc06640 10 monclient(hunting): picked mon.noname-c con 0x7f0588140f80 addr v1:172.21.15.98:6790/0 2024-03-29T11:48:19.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f058dc06640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05881423a0 con 0x7f0588140660 2024-03-29T11:48:19.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f058dc06640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0588140140 con 0x7f058809ac40 2024-03-29T11:48:19.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f058dc06640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05880b8040 con 0x7f0588140f80 2024-03-29T11:48:19.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f058dc06640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:19.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f058dc06640 10 monclient(hunting): authenticate will time out at 809.767327s 2024-03-29T11:48:19.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f05877fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f058809ac40 0x7f05881d7480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:51504/0 (socket says 172.21.15.121:51504) 2024-03-29T11:48:19.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f05877fe640 1 -- 172.21.15.121:0/270939538 learned_addr learned my addr 172.21.15.121:0/270939538 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-29T11:48:19.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 10 client.?.objecter ms_handle_connect 0x7f058809ac40 2024-03-29T11:48:19.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:19.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 10 client.?.objecter ms_handle_connect 0x7f0588140660 2024-03-29T11:48:19.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:19.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 10 client.?.objecter ms_handle_connect 0x7f0588140f80 2024-03-29T11:48:19.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:19.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 1 -- 172.21.15.121:0/270939538 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 441304795 0 0) 0x7f0588140140 con 0x7f058809ac40 2024-03-29T11:48:19.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:48:19.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:19.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:19.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 10 monclient(hunting): my global_id is 4150 2024-03-29T11:48:19.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 1 -- 172.21.15.121:0/270939538 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f056c004eb0 con 0x7f058809ac40 2024-03-29T11:48:19.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 1 -- 172.21.15.121:0/270939538 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 282690945 0 0) 0x7f05881423a0 con 0x7f0588140660 2024-03-29T11:48:19.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:48:19.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:19.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:19.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 10 monclient(hunting): my global_id is 4143 2024-03-29T11:48:19.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 1 -- 172.21.15.121:0/270939538 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0588140140 con 0x7f0588140660 2024-03-29T11:48:19.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 1 -- 172.21.15.121:0/270939538 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 735762633 0 0) 0x7f05880b8040 con 0x7f0588140f80 2024-03-29T11:48:19.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:48:19.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:19.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:19.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 10 monclient(hunting): my global_id is 4142 2024-03-29T11:48:19.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 1 -- 172.21.15.121:0/270939538 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05881423a0 con 0x7f0588140f80 2024-03-29T11:48:19.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 1 -- 172.21.15.121:0/270939538 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 346806043 0 0) 0x7f056c004eb0 con 0x7f058809ac40 2024-03-29T11:48:19.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 1 -- 172.21.15.121:0/270939538 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05880b8040 con 0x7f058809ac40 2024-03-29T11:48:19.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 1 -- 172.21.15.121:0/270939538 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f05780019b0 con 0x7f058809ac40 2024-03-29T11:48:19.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:19.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:48:19.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 10 monclient(hunting): dump: 2024-03-29T11:48:19.617 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:19.617 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:19.617 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.617 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.617 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:19.617 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:19.617 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:19.617 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:19.617 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:19.618 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:19.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:48:19.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 1 -- 172.21.15.121:0/270939538 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1394159996 0 0) 0x7f0588140140 con 0x7f0588140660 2024-03-29T11:48:19.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 1 -- 172.21.15.121:0/270939538 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f056c004eb0 con 0x7f0588140660 2024-03-29T11:48:19.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 1 -- 172.21.15.121:0/270939538 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f0570001b80 con 0x7f0588140660 2024-03-29T11:48:19.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:19.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:48:19.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 10 monclient(hunting): dump: 2024-03-29T11:48:19.618 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:19.618 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:19.618 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.618 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.618 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:19.618 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:19.619 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:19.619 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:19.619 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:19.619 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:19.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 1 -- 172.21.15.121:0/270939538 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2687892154 0 0) 0x7f05881423a0 con 0x7f0588140f80 2024-03-29T11:48:19.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 1 -- 172.21.15.121:0/270939538 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0588140140 con 0x7f0588140f80 2024-03-29T11:48:19.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 1 -- 172.21.15.121:0/270939538 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f0580001b40 con 0x7f0588140f80 2024-03-29T11:48:19.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:19.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:48:19.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 10 monclient(hunting): dump: 2024-03-29T11:48:19.619 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:19.619 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:19.619 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.619 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.620 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:19.620 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:19.620 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:19.620 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:19.620 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:19.620 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:19.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 1 -- 172.21.15.121:0/270939538 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2497977585 0 0) 0x7f05880b8040 con 0x7f058809ac40 2024-03-29T11:48:19.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 1 -- 172.21.15.121:0/270939538 >> v1:172.21.15.98:6790/0 conn(0x7f0588140f80 legacy=0x7f05881e2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:19.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 1 -- 172.21.15.121:0/270939538 >> v1:172.21.15.98:6789/0 conn(0x7f0588140660 legacy=0x7f05880b1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:19.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:48:19.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 1 monclient: found mon.b 2024-03-29T11:48:19.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:19.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:19.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 1 -- 172.21.15.121:0/270939538 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05880b81d0 con 0x7f058809ac40 2024-03-29T11:48:19.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 10 monclient: _finish_auth 0 2024-03-29T11:48:19.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:48:19.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:19.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f058dc06640 5 monclient: authenticate success, global_id 4150 2024-03-29T11:48:19.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f058dc06640 10 monclient: _renew_subs 2024-03-29T11:48:19.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f058dc06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:19.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f058dc06640 1 -- 172.21.15.121:0/270939538 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05881d7b90 con 0x7f058809ac40 2024-03-29T11:48:19.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f058dc06640 10 client.4150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:48:19.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f058dc06640 10 monclient: _renew_subs 2024-03-29T11:48:19.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f058dc06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:19.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f058dc06640 1 -- 172.21.15.121:0/270939538 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f05880b19f0 con 0x7f058809ac40 2024-03-29T11:48:19.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f058dc06640 1 librados: init done 2024-03-29T11:48:19.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 1 -- 172.21.15.121:0/270939538 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0578001c50 con 0x7f058809ac40 2024-03-29T11:48:19.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:19.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 1 -- 172.21.15.121:0/270939538 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f0578003a60 con 0x7f058809ac40 2024-03-29T11:48:19.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.597+0000 7f0584ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:19.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.598+0000 7f0584ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:48:19.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.598+0000 7f0584ff9640 10 monclient: dump: 2024-03-29T11:48:19.622 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:19.622 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:19.622 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.622 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.622 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:19.622 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:19.622 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:19.622 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:19.622 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:19.622 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:19.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.598+0000 7f05867fc640 4 set_mon_vals no callback set 2024-03-29T11:48:19.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.598+0000 7f0584ff9640 1 -- 172.21.15.121:0/270939538 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0578004010 con 0x7f058809ac40 2024-03-29T11:48:19.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.598+0000 7f0584ff9640 1 -- 172.21.15.121:0/270939538 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 2550198626 0 0) 0x7f0578006880 con 0x7f058809ac40 2024-03-29T11:48:19.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.598+0000 7f0584ff9640 10 client.4150.objecter ms_dispatch 0x7f0588000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T11:48:19.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.598+0000 7f0584ff9640 3 client.4150.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T11:48:19.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.598+0000 7f0584ff9640 3 client.4150.objecter handle_osd_map decoding full epoch 7 2024-03-29T11:48:19.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.598+0000 7f0584ff9640 20 client.4150.objecter dump_active .. 0 homeless 2024-03-29T11:48:19.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.598+0000 7f058dc06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:19.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.598+0000 7f058dc06640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:19.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.598+0000 7f058dc06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:19.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.598+0000 7f058dc06640 1 -- 172.21.15.121:0/270939538 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0560005080 con 0x7f058809ac40 2024-03-29T11:48:19.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.600+0000 7f0584ff9640 1 -- 172.21.15.121:0/270939538 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0578006b60 con 0x7f058809ac40 2024-03-29T11:48:19.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.600+0000 7f0584ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:19.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.600+0000 7f0584ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:48:19.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.672+0000 7f058dc06640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "510d2deb-540c-43f6-8d32-6783917c2208", "id": 5}] 2024-03-29T11:48:19.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.673+0000 7f058dc06640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "510d2deb-540c-43f6-8d32-6783917c2208", "id": 5}] 2024-03-29T11:48:19.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.673+0000 7f058dc06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:19.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.673+0000 7f058dc06640 1 -- 172.21.15.121:0/270939538 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd new", "uuid": "510d2deb-540c-43f6-8d32-6783917c2208", "id": 5} v 0) v1 -- 0x7f05600058b0 con 0x7f058809ac40 2024-03-29T11:48:19.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.676+0000 7f0584ff9640 1 -- 172.21.15.121:0/270939538 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "510d2deb-540c-43f6-8d32-6783917c2208", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 1851212456 0 1988412176) 0x7f0578006300 con 0x7f058809ac40 2024-03-29T11:48:19.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.676+0000 7f0584ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "510d2deb-540c-43f6-8d32-6783917c2208", "id": 5}] 2024-03-29T11:48:19.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.676+0000 7f0584ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:48:19.682 INFO:teuthology.orchestra.run.smithi121.stdout:5 2024-03-29T11:48:19.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.677+0000 7f058dc06640 10 librados: watch_flush enter 2024-03-29T11:48:19.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.677+0000 7f058dc06640 10 librados: watch_flush exit 2024-03-29T11:48:19.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.677+0000 7f058dc06640 20 client.4150.objecter shutdown clearing up homeless session... 2024-03-29T11:48:19.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.677+0000 7f058dc06640 10 client.4150.objecter successfully canceled tick 2024-03-29T11:48:19.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.677+0000 7f058dc06640 10 monclient: shutdown 2024-03-29T11:48:19.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.677+0000 7f058dc06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:19.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.677+0000 7f058dc06640 1 -- 172.21.15.121:0/270939538 >> v1:172.21.15.121:6789/0 conn(0x7f058809ac40 legacy=0x7f05881d7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:19.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.677+0000 7f058dc06640 1 -- 172.21.15.121:0/270939538 shutdown_connections 2024-03-29T11:48:19.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.677+0000 7f058dc06640 1 -- 172.21.15.121:0/270939538 >> 172.21.15.121:0/270939538 conn(0x7f0588066b00 msgr2=0x7f058813c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:19.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.678+0000 7f058dc06640 1 -- 172.21.15.121:0/270939538 shutdown_connections 2024-03-29T11:48:19.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.678+0000 7f058dc06640 1 -- 172.21.15.121:0/270939538 wait complete. 2024-03-29T11:48:19.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.678+0000 7f058dc06640 1 librados: shutdown 2024-03-29T11:48:19.693 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new bcd19919-e170-47a2-8b81-bcd5931571fd 6 2024-03-29T11:48:19.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.787+0000 7f809832b640 10 monclient: get_monmap_and_config 2024-03-29T11:48:19.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.787+0000 7f809832b640 10 monclient: build_initial_monmap 2024-03-29T11:48:19.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.787+0000 7f809832b640 10 monclient: monmap: 2024-03-29T11:48:19.794 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-29T11:48:19.794 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:19.794 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:48:19.788840+0000 2024-03-29T11:48:19.794 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:48:19.788840+0000 2024-03-29T11:48:19.794 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:19.794 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:19.794 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:48:19.794 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:48:19.794 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:48:19.794 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:19.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.788+0000 7f809832b640 1 Processor -- start 2024-03-29T11:48:19.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.788+0000 7f809832b640 1 -- start start 2024-03-29T11:48:19.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.788+0000 7f809832b640 10 monclient: init 2024-03-29T11:48:19.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f809832b640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:19.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f809832b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:48:19.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f809832b640 10 monclient(hunting): picked mon.noname-a con 0x7f809009ac40 addr v1:172.21.15.98:6789/0 2024-03-29T11:48:19.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f809832b640 10 monclient(hunting): picked mon.noname-c con 0x7f809009c870 addr v1:172.21.15.121:6789/0 2024-03-29T11:48:19.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f809832b640 10 monclient(hunting): picked mon.noname-b con 0x7f80900b4360 addr v1:172.21.15.98:6790/0 2024-03-29T11:48:19.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f809832b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f809009dec0 con 0x7f809009ac40 2024-03-29T11:48:19.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f809832b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f809009da30 con 0x7f809009c870 2024-03-29T11:48:19.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f809832b640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80900b39c0 con 0x7f80900b4360 2024-03-29T11:48:19.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f809832b640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:19.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f809832b640 10 monclient(hunting): authenticate will time out at 809.959442s 2024-03-29T11:48:19.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f80958c6640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f809009c870 0x7f809013f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:51506/0 (socket says 172.21.15.121:51506) 2024-03-29T11:48:19.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f80958c6640 1 -- 172.21.15.121:0/2215415285 learned_addr learned my addr 172.21.15.121:0/2215415285 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-29T11:48:19.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f80950c5640 1 -- 172.21.15.121:0/2215415285 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2855725651 0 0) 0x7f809009da30 con 0x7f809009c870 2024-03-29T11:48:19.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f80950c5640 10 monclient(hunting): _negotiate 2024-03-29T11:48:19.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f80950c5640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:19.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f80950c5640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:19.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f80950c5640 10 monclient(hunting): my global_id is 4153 2024-03-29T11:48:19.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f80950c5640 1 -- 172.21.15.121:0/2215415285 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f807c003520 con 0x7f809009c870 2024-03-29T11:48:19.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f80950c5640 1 -- 172.21.15.121:0/2215415285 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3287683747 0 0) 0x7f8080001110 con 0x7f809009ac40 2024-03-29T11:48:19.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f80950c5640 10 monclient(hunting): _negotiate 2024-03-29T11:48:19.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f80950c5640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:19.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f80950c5640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:19.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f80950c5640 10 monclient(hunting): my global_id is 4146 2024-03-29T11:48:19.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f80950c5640 1 -- 172.21.15.121:0/2215415285 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f809009da30 con 0x7f809009ac40 2024-03-29T11:48:19.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f80950c5640 1 -- 172.21.15.121:0/2215415285 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3245880194 0 0) 0x7f80900b39c0 con 0x7f80900b4360 2024-03-29T11:48:19.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f80950c5640 10 monclient(hunting): _negotiate 2024-03-29T11:48:19.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f80950c5640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:19.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f80950c5640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:19.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f80950c5640 10 monclient(hunting): my global_id is 4145 2024-03-29T11:48:19.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f80950c5640 1 -- 172.21.15.121:0/2215415285 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8080001110 con 0x7f80900b4360 2024-03-29T11:48:19.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f80950c5640 1 -- 172.21.15.121:0/2215415285 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2621020843 0 0) 0x7f807c003520 con 0x7f809009c870 2024-03-29T11:48:19.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f80950c5640 1 -- 172.21.15.121:0/2215415285 >> v1:172.21.15.98:6790/0 conn(0x7f80900b4360 legacy=0x7f809014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:19.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f80950c5640 1 -- 172.21.15.121:0/2215415285 >> v1:172.21.15.98:6789/0 conn(0x7f809009ac40 legacy=0x7f80900b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:19.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f80950c5640 10 monclient: _finish_hunting 0 2024-03-29T11:48:19.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f80950c5640 1 monclient: found mon.noname-c 2024-03-29T11:48:19.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f80950c5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:19.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f80950c5640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:48:19.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f80950c5640 1 -- 172.21.15.121:0/2215415285 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80900b5160 con 0x7f809009c870 2024-03-29T11:48:19.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f80950c5640 10 monclient: _finish_auth 0 2024-03-29T11:48:19.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f80950c5640 10 monclient: _check_auth_tickets 2024-03-29T11:48:19.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f80950c5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:19.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f80950c5640 1 -- 172.21.15.121:0/2215415285 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f808c001690 con 0x7f809009c870 2024-03-29T11:48:19.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f809832b640 5 monclient: authenticate success, global_id 4153 2024-03-29T11:48:19.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f809832b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:48:19.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f80950c5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:19.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f80950c5640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:48:19.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f80950c5640 10 monclient: dump: 2024-03-29T11:48:19.806 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:19.806 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:19.806 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.806 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.806 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:19.806 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:19.806 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:19.806 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:19.806 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:19.806 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:19.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f809832b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:48:19.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f80950c5640 1 -- 172.21.15.121:0/2215415285 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f808c001870 con 0x7f809009c870 2024-03-29T11:48:19.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f80950c5640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:19.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f80950c5640 1 -- 172.21.15.121:0/2215415285 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f808c001c80 con 0x7f809009c870 2024-03-29T11:48:19.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f809832b640 10 monclient: get_monmap_and_config success 2024-03-29T11:48:19.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f809832b640 4 set_mon_vals no callback set 2024-03-29T11:48:19.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f80950c5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:19.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f80950c5640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:48:19.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.789+0000 7f80950c5640 10 monclient: dump: 2024-03-29T11:48:19.807 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:19.807 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:19.807 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.807 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.807 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:19.807 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:19.807 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:19.808 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.790+0000 7f809832b640 10 monclient: shutdown 2024-03-29T11:48:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.790+0000 7f809832b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.790+0000 7f809832b640 1 -- 172.21.15.121:0/2215415285 >> v1:172.21.15.121:6789/0 conn(0x7f809009c870 legacy=0x7f809013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.790+0000 7f809832b640 1 -- 172.21.15.121:0/2215415285 shutdown_connections 2024-03-29T11:48:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.790+0000 7f809832b640 1 -- 172.21.15.121:0/2215415285 >> 172.21.15.121:0/2215415285 conn(0x7f8090066b00 msgr2=0x7f8090066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.790+0000 7f809832b640 1 -- 172.21.15.121:0/2215415285 shutdown_connections 2024-03-29T11:48:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.790+0000 7f809832b640 1 -- 172.21.15.121:0/2215415285 wait complete. 2024-03-29T11:48:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.791+0000 7f809832b640 10 monclient: build_initial_monmap 2024-03-29T11:48:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.791+0000 7f809832b640 10 monclient: monmap: 2024-03-29T11:48:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-29T11:48:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:48:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-03-29T11:48:19.809 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-03-29T11:48:19.809 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:19.809 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:19.809 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:48:19.809 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:48:19.809 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:48:19.809 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:19.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.791+0000 7f809832b640 1 librados: starting msgr at 2024-03-29T11:48:19.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.791+0000 7f809832b640 1 librados: starting objecter 2024-03-29T11:48:19.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.791+0000 7f809832b640 1 Processor -- start 2024-03-29T11:48:19.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.791+0000 7f809832b640 1 -- start start 2024-03-29T11:48:19.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.791+0000 7f809832b640 1 librados: setting wanted keys 2024-03-29T11:48:19.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.791+0000 7f809832b640 1 librados: calling monclient init 2024-03-29T11:48:19.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.791+0000 7f809832b640 10 monclient: init 2024-03-29T11:48:19.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f809832b640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:19.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f809832b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:48:19.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f809832b640 10 monclient(hunting): picked mon.noname-b con 0x7f809009ac40 addr v1:172.21.15.121:6789/0 2024-03-29T11:48:19.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f809832b640 10 monclient(hunting): picked mon.noname-a con 0x7f8090138df0 addr v1:172.21.15.98:6789/0 2024-03-29T11:48:19.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f809832b640 10 monclient(hunting): picked mon.noname-c con 0x7f80900b05c0 addr v1:172.21.15.98:6790/0 2024-03-29T11:48:19.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f809832b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80900afe90 con 0x7f8090138df0 2024-03-29T11:48:19.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f809832b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80900b1920 con 0x7f809009ac40 2024-03-29T11:48:19.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f809832b640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80900b0a10 con 0x7f80900b05c0 2024-03-29T11:48:19.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f809832b640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:19.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f809832b640 10 monclient(hunting): authenticate will time out at 809.962352s 2024-03-29T11:48:19.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f80960c7640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f809009ac40 0x7f80901d7580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:51516/0 (socket says 172.21.15.121:51516) 2024-03-29T11:48:19.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f80960c7640 1 -- 172.21.15.121:0/2043138269 learned_addr learned my addr 172.21.15.121:0/2043138269 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-29T11:48:19.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 10 client.?.objecter ms_handle_connect 0x7f809009ac40 2024-03-29T11:48:19.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:19.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 10 client.?.objecter ms_handle_connect 0x7f8090138df0 2024-03-29T11:48:19.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:19.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 10 client.?.objecter ms_handle_connect 0x7f80900b05c0 2024-03-29T11:48:19.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:19.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 1 -- 172.21.15.121:0/2043138269 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 429570182 0 0) 0x7f80900b1920 con 0x7f809009ac40 2024-03-29T11:48:19.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:48:19.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:19.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:19.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 10 monclient(hunting): my global_id is 4156 2024-03-29T11:48:19.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 1 -- 172.21.15.121:0/2043138269 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8074004eb0 con 0x7f809009ac40 2024-03-29T11:48:19.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 1 -- 172.21.15.121:0/2043138269 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2648017023 0 0) 0x7f80900b0a10 con 0x7f80900b05c0 2024-03-29T11:48:19.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:48:19.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:19.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:19.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 10 monclient(hunting): my global_id is 4148 2024-03-29T11:48:19.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 1 -- 172.21.15.121:0/2043138269 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80900b1920 con 0x7f80900b05c0 2024-03-29T11:48:19.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 1 -- 172.21.15.121:0/2043138269 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1501703496 0 0) 0x7f80900afe90 con 0x7f8090138df0 2024-03-29T11:48:19.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:48:19.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:19.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:19.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 10 monclient(hunting): my global_id is 4149 2024-03-29T11:48:19.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 1 -- 172.21.15.121:0/2043138269 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80900b0a10 con 0x7f8090138df0 2024-03-29T11:48:19.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 1 -- 172.21.15.121:0/2043138269 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2950110221 0 0) 0x7f8074004eb0 con 0x7f809009ac40 2024-03-29T11:48:19.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 1 -- 172.21.15.121:0/2043138269 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80900afe90 con 0x7f809009ac40 2024-03-29T11:48:19.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 1 -- 172.21.15.121:0/2043138269 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f80800019a0 con 0x7f809009ac40 2024-03-29T11:48:19.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:19.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:48:19.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 10 monclient(hunting): dump: 2024-03-29T11:48:19.812 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:19.812 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:19.812 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.813 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.813 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:19.813 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:19.813 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:19.813 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:19.813 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:19.813 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:19.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:48:19.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 1 -- 172.21.15.121:0/2043138269 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1831929933 0 0) 0x7f80900b1920 con 0x7f80900b05c0 2024-03-29T11:48:19.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 1 -- 172.21.15.121:0/2043138269 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8074004eb0 con 0x7f80900b05c0 2024-03-29T11:48:19.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 1 -- 172.21.15.121:0/2043138269 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f8088001a40 con 0x7f80900b05c0 2024-03-29T11:48:19.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:19.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:48:19.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 10 monclient(hunting): dump: 2024-03-29T11:48:19.813 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:19.813 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:19.814 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.814 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.814 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:19.814 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:19.814 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:19.814 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:19.814 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:19.814 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:19.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 1 -- 172.21.15.121:0/2043138269 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2042641860 0 0) 0x7f80900b0a10 con 0x7f8090138df0 2024-03-29T11:48:19.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 1 -- 172.21.15.121:0/2043138269 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80900b1920 con 0x7f8090138df0 2024-03-29T11:48:19.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 1 -- 172.21.15.121:0/2043138269 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f808c0019e0 con 0x7f8090138df0 2024-03-29T11:48:19.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:19.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:48:19.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 10 monclient(hunting): dump: 2024-03-29T11:48:19.815 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:19.815 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:19.815 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.815 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.815 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:19.815 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:19.815 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:19.815 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:19.815 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:19.815 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:19.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 1 -- 172.21.15.121:0/2043138269 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 603873405 0 0) 0x7f80900afe90 con 0x7f809009ac40 2024-03-29T11:48:19.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 1 -- 172.21.15.121:0/2043138269 >> v1:172.21.15.98:6790/0 conn(0x7f80900b05c0 legacy=0x7f8090138100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:19.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 1 -- 172.21.15.121:0/2043138269 >> v1:172.21.15.98:6789/0 conn(0x7f8090138df0 legacy=0x7f80900af780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:19.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:48:19.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 1 monclient: found mon.b 2024-03-29T11:48:19.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:19.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:19.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 1 -- 172.21.15.121:0/2043138269 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80900b00e0 con 0x7f809009ac40 2024-03-29T11:48:19.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 10 monclient: _finish_auth 0 2024-03-29T11:48:19.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:48:19.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:19.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f809832b640 5 monclient: authenticate success, global_id 4156 2024-03-29T11:48:19.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f809832b640 10 monclient: _renew_subs 2024-03-29T11:48:19.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f809832b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:19.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f809832b640 1 -- 172.21.15.121:0/2043138269 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8090139890 con 0x7f809009ac40 2024-03-29T11:48:19.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f809832b640 10 client.4156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:48:19.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f809832b640 10 monclient: _renew_subs 2024-03-29T11:48:19.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f809832b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:19.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f809832b640 1 -- 172.21.15.121:0/2043138269 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f80901389f0 con 0x7f809009ac40 2024-03-29T11:48:19.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f809832b640 1 librados: init done 2024-03-29T11:48:19.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.792+0000 7f8086ffd640 1 -- 172.21.15.121:0/2043138269 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8080001c40 con 0x7f809009ac40 2024-03-29T11:48:19.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.793+0000 7f8086ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:19.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.793+0000 7f8086ffd640 1 -- 172.21.15.121:0/2043138269 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f8080003a60 con 0x7f809009ac40 2024-03-29T11:48:19.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.793+0000 7f8086ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:19.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.793+0000 7f8086ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:48:19.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.793+0000 7f80948c4640 4 set_mon_vals no callback set 2024-03-29T11:48:19.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.793+0000 7f8086ffd640 10 monclient: dump: 2024-03-29T11:48:19.817 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:19.817 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:19.817 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.817 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:19.817 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:19.817 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:19.817 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:19.817 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:19.817 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:19.818 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:19.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.793+0000 7f8086ffd640 1 -- 172.21.15.121:0/2043138269 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8080004010 con 0x7f809009ac40 2024-03-29T11:48:19.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.793+0000 7f8086ffd640 1 -- 172.21.15.121:0/2043138269 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 4191327164 0 0) 0x7f8080006820 con 0x7f809009ac40 2024-03-29T11:48:19.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.793+0000 7f8086ffd640 10 client.4156.objecter ms_dispatch 0x7f8090000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T11:48:19.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.793+0000 7f8086ffd640 3 client.4156.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T11:48:19.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.793+0000 7f8086ffd640 3 client.4156.objecter handle_osd_map decoding full epoch 8 2024-03-29T11:48:19.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.793+0000 7f8086ffd640 20 client.4156.objecter dump_active .. 0 homeless 2024-03-29T11:48:19.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.793+0000 7f809832b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:19.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.793+0000 7f809832b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:19.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.793+0000 7f809832b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:19.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.793+0000 7f809832b640 1 -- 172.21.15.121:0/2043138269 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8078005080 con 0x7f809009ac40 2024-03-29T11:48:19.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.795+0000 7f8086ffd640 1 -- 172.21.15.121:0/2043138269 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f8080006230 con 0x7f809009ac40 2024-03-29T11:48:19.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.795+0000 7f8086ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:19.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.795+0000 7f8086ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:48:19.843 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T11:48:19.837+0000 7faaea042040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:48:19.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.868+0000 7f809832b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "bcd19919-e170-47a2-8b81-bcd5931571fd", "id": 6}] 2024-03-29T11:48:19.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.868+0000 7f809832b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "bcd19919-e170-47a2-8b81-bcd5931571fd", "id": 6}] 2024-03-29T11:48:19.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.868+0000 7f809832b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:19.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.868+0000 7f809832b640 1 -- 172.21.15.121:0/2043138269 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd new", "uuid": "bcd19919-e170-47a2-8b81-bcd5931571fd", "id": 6} v 0) v1 -- 0x7f80780058b0 con 0x7f809009ac40 2024-03-29T11:48:19.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.872+0000 7f8086ffd640 1 -- 172.21.15.121:0/2043138269 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bcd19919-e170-47a2-8b81-bcd5931571fd", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 2712378355 0 1113811849) 0x7f80800064d0 con 0x7f809009ac40 2024-03-29T11:48:19.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.872+0000 7f8086ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "bcd19919-e170-47a2-8b81-bcd5931571fd", "id": 6}] 2024-03-29T11:48:19.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.872+0000 7f8086ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:48:19.878 INFO:teuthology.orchestra.run.smithi121.stdout:6 2024-03-29T11:48:19.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.873+0000 7f809832b640 10 librados: watch_flush enter 2024-03-29T11:48:19.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.873+0000 7f809832b640 10 librados: watch_flush exit 2024-03-29T11:48:19.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.873+0000 7f809832b640 20 client.4156.objecter shutdown clearing up homeless session... 2024-03-29T11:48:19.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.873+0000 7f809832b640 10 client.4156.objecter successfully canceled tick 2024-03-29T11:48:19.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.873+0000 7f809832b640 10 monclient: shutdown 2024-03-29T11:48:19.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.873+0000 7f809832b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:19.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.873+0000 7f809832b640 1 -- 172.21.15.121:0/2043138269 >> v1:172.21.15.121:6789/0 conn(0x7f809009ac40 legacy=0x7f80901d7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:19.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.873+0000 7f809832b640 1 -- 172.21.15.121:0/2043138269 shutdown_connections 2024-03-29T11:48:19.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.873+0000 7f809832b640 1 -- 172.21.15.121:0/2043138269 >> 172.21.15.121:0/2043138269 conn(0x7f8090066b00 msgr2=0x7f809013d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:19.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.873+0000 7f809832b640 1 -- 172.21.15.121:0/2043138269 shutdown_connections 2024-03-29T11:48:19.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.873+0000 7f809832b640 1 -- 172.21.15.121:0/2043138269 wait complete. 2024-03-29T11:48:19.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.873+0000 7f809832b640 1 librados: shutdown 2024-03-29T11:48:19.889 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 48b0600d-02cc-4677-9176-be8248744f5e 7 2024-03-29T11:48:19.950 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:48:19.944+0000 7f4e33c10040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:48:19.978 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T11:48:19.973+0000 7faaea042040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:48:19.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.983+0000 7f64ac6a8640 10 monclient: get_monmap_and_config 2024-03-29T11:48:19.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.983+0000 7f64ac6a8640 10 monclient: build_initial_monmap 2024-03-29T11:48:19.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.983+0000 7f64ac6a8640 10 monclient: monmap: 2024-03-29T11:48:19.989 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-29T11:48:19.989 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:19.989 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:48:19.984161+0000 2024-03-29T11:48:19.989 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:48:19.984161+0000 2024-03-29T11:48:19.989 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:19.989 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:19.989 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:48:19.989 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:48:19.989 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:48:19.989 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:19.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.983+0000 7f64ac6a8640 1 Processor -- start 2024-03-29T11:48:19.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.983+0000 7f64ac6a8640 1 -- start start 2024-03-29T11:48:19.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.983+0000 7f64ac6a8640 10 monclient: init 2024-03-29T11:48:19.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.984+0000 7f64ac6a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:19.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.984+0000 7f64ac6a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:48:19.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.984+0000 7f64ac6a8640 10 monclient(hunting): picked mon.noname-a con 0x7f64a409ac40 addr v1:172.21.15.98:6789/0 2024-03-29T11:48:19.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.984+0000 7f64ac6a8640 10 monclient(hunting): picked mon.noname-c con 0x7f64a409c870 addr v1:172.21.15.121:6789/0 2024-03-29T11:48:19.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.984+0000 7f64ac6a8640 10 monclient(hunting): picked mon.noname-b con 0x7f64a40b4360 addr v1:172.21.15.98:6790/0 2024-03-29T11:48:19.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.984+0000 7f64ac6a8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64a409dec0 con 0x7f64a409ac40 2024-03-29T11:48:19.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.984+0000 7f64ac6a8640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64a409da30 con 0x7f64a409c870 2024-03-29T11:48:19.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.984+0000 7f64ac6a8640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64a40b39c0 con 0x7f64a40b4360 2024-03-29T11:48:19.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.984+0000 7f64ac6a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:19.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.984+0000 7f64ac6a8640 10 monclient(hunting): authenticate will time out at 810.154753s 2024-03-29T11:48:19.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.984+0000 7f64a9c43640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f64a409c870 0x7f64a413f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:51524/0 (socket says 172.21.15.121:51524) 2024-03-29T11:48:19.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.984+0000 7f64a9c43640 1 -- 172.21.15.121:0/1326625523 learned_addr learned my addr 172.21.15.121:0/1326625523 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-29T11:48:19.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.984+0000 7f64a9442640 1 -- 172.21.15.121:0/1326625523 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 902562169 0 0) 0x7f64a409da30 con 0x7f64a409c870 2024-03-29T11:48:19.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.984+0000 7f64a9442640 10 monclient(hunting): _negotiate 2024-03-29T11:48:19.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.984+0000 7f64a9442640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:19.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.984+0000 7f64a9442640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:20.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.984+0000 7f64a9442640 10 monclient(hunting): my global_id is 4159 2024-03-29T11:48:20.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.984+0000 7f64a9442640 1 -- 172.21.15.121:0/1326625523 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6490003520 con 0x7f64a409c870 2024-03-29T11:48:20.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.984+0000 7f64a9442640 1 -- 172.21.15.121:0/1326625523 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3314822563 0 0) 0x7f64a40b39c0 con 0x7f64a40b4360 2024-03-29T11:48:20.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.984+0000 7f64a9442640 10 monclient(hunting): _negotiate 2024-03-29T11:48:20.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.984+0000 7f64a9442640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:20.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.984+0000 7f64a9442640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:20.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.984+0000 7f64a9442640 10 monclient(hunting): my global_id is 4151 2024-03-29T11:48:20.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.984+0000 7f64a9442640 1 -- 172.21.15.121:0/1326625523 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64a409da30 con 0x7f64a40b4360 2024-03-29T11:48:20.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.984+0000 7f64a9442640 1 -- 172.21.15.121:0/1326625523 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 561535729 0 0) 0x7f64a0001040 con 0x7f64a409ac40 2024-03-29T11:48:20.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.984+0000 7f64a9442640 10 monclient(hunting): _negotiate 2024-03-29T11:48:20.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.984+0000 7f64a9442640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:20.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.984+0000 7f64a9442640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:20.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.984+0000 7f64a9442640 10 monclient(hunting): my global_id is 4152 2024-03-29T11:48:20.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.984+0000 7f64a9442640 1 -- 172.21.15.121:0/1326625523 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64a40b39c0 con 0x7f64a409ac40 2024-03-29T11:48:20.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.985+0000 7f64a9442640 1 -- 172.21.15.121:0/1326625523 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1223315359 0 0) 0x7f6490003520 con 0x7f64a409c870 2024-03-29T11:48:20.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.985+0000 7f64a9442640 1 -- 172.21.15.121:0/1326625523 >> v1:172.21.15.98:6790/0 conn(0x7f64a40b4360 legacy=0x7f64a414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:20.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.985+0000 7f64a9442640 1 -- 172.21.15.121:0/1326625523 >> v1:172.21.15.98:6789/0 conn(0x7f64a409ac40 legacy=0x7f64a40b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:20.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.985+0000 7f64a9442640 10 monclient: _finish_hunting 0 2024-03-29T11:48:20.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.985+0000 7f64a9442640 1 monclient: found mon.noname-c 2024-03-29T11:48:20.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.985+0000 7f64a9442640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:20.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.985+0000 7f64a9442640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:48:20.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.985+0000 7f64a9442640 1 -- 172.21.15.121:0/1326625523 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a40b5160 con 0x7f64a409c870 2024-03-29T11:48:20.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.985+0000 7f64a9442640 10 monclient: _finish_auth 0 2024-03-29T11:48:20.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.985+0000 7f64a9442640 10 monclient: _check_auth_tickets 2024-03-29T11:48:20.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.985+0000 7f64a9442640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:20.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.985+0000 7f64a9442640 1 -- 172.21.15.121:0/1326625523 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f6494001690 con 0x7f64a409c870 2024-03-29T11:48:20.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.985+0000 7f64ac6a8640 5 monclient: authenticate success, global_id 4159 2024-03-29T11:48:20.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.985+0000 7f64ac6a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:48:20.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.985+0000 7f64a9442640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:20.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.985+0000 7f64a9442640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:48:20.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.985+0000 7f64a9442640 10 monclient: dump: 2024-03-29T11:48:20.003 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:20.003 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:20.003 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:20.003 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:20.003 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:20.003 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:20.003 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:20.003 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:20.003 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:20.003 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:20.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.985+0000 7f64ac6a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:48:20.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.985+0000 7f64a9442640 1 -- 172.21.15.121:0/1326625523 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6494001870 con 0x7f64a409c870 2024-03-29T11:48:20.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.985+0000 7f64a9442640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:20.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.985+0000 7f64a9442640 1 -- 172.21.15.121:0/1326625523 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f6494001c80 con 0x7f64a409c870 2024-03-29T11:48:20.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.985+0000 7f64ac6a8640 10 monclient: get_monmap_and_config success 2024-03-29T11:48:20.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.985+0000 7f64ac6a8640 4 set_mon_vals no callback set 2024-03-29T11:48:20.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.985+0000 7f64a9442640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:20.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.985+0000 7f64a9442640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:48:20.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.985+0000 7f64a9442640 10 monclient: dump: 2024-03-29T11:48:20.004 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:20.004 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:20.004 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:20.004 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:20.005 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:20.005 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:20.005 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:20.005 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:20.005 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:20.005 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:20.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.985+0000 7f64ac6a8640 10 monclient: shutdown 2024-03-29T11:48:20.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.985+0000 7f64ac6a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:20.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.985+0000 7f64ac6a8640 1 -- 172.21.15.121:0/1326625523 >> v1:172.21.15.121:6789/0 conn(0x7f64a409c870 legacy=0x7f64a413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:20.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.985+0000 7f64ac6a8640 1 -- 172.21.15.121:0/1326625523 shutdown_connections 2024-03-29T11:48:20.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.985+0000 7f64ac6a8640 1 -- 172.21.15.121:0/1326625523 >> 172.21.15.121:0/1326625523 conn(0x7f64a4066b00 msgr2=0x7f64a4066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:20.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.985+0000 7f64ac6a8640 1 -- 172.21.15.121:0/1326625523 shutdown_connections 2024-03-29T11:48:20.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.985+0000 7f64ac6a8640 1 -- 172.21.15.121:0/1326625523 wait complete. 2024-03-29T11:48:20.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.986+0000 7f64ac6a8640 10 monclient: build_initial_monmap 2024-03-29T11:48:20.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.986+0000 7f64ac6a8640 10 monclient: monmap: 2024-03-29T11:48:20.006 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-29T11:48:20.006 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:48:20.006 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-03-29T11:48:20.006 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-03-29T11:48:20.006 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:20.006 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:20.007 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:48:20.007 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:48:20.007 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:48:20.007 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:20.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.986+0000 7f64ac6a8640 1 librados: starting msgr at 2024-03-29T11:48:20.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.986+0000 7f64ac6a8640 1 librados: starting objecter 2024-03-29T11:48:20.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.986+0000 7f64ac6a8640 1 Processor -- start 2024-03-29T11:48:20.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.986+0000 7f64ac6a8640 1 -- start start 2024-03-29T11:48:20.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.986+0000 7f64ac6a8640 1 librados: setting wanted keys 2024-03-29T11:48:20.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.986+0000 7f64ac6a8640 1 librados: calling monclient init 2024-03-29T11:48:20.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.986+0000 7f64ac6a8640 10 monclient: init 2024-03-29T11:48:20.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.987+0000 7f64ac6a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:20.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.987+0000 7f64ac6a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:48:20.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.987+0000 7f64ac6a8640 10 monclient(hunting): picked mon.noname-a con 0x7f64a409ac40 addr v1:172.21.15.98:6789/0 2024-03-29T11:48:20.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.987+0000 7f64ac6a8640 10 monclient(hunting): picked mon.noname-b con 0x7f64a40b09f0 addr v1:172.21.15.121:6789/0 2024-03-29T11:48:20.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.987+0000 7f64ac6a8640 10 monclient(hunting): picked mon.noname-c con 0x7f64a41d7f10 addr v1:172.21.15.98:6790/0 2024-03-29T11:48:20.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.987+0000 7f64ac6a8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64a40b19f0 con 0x7f64a409ac40 2024-03-29T11:48:20.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.987+0000 7f64ac6a8640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64a41d72e0 con 0x7f64a40b09f0 2024-03-29T11:48:20.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.987+0000 7f64ac6a8640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64a40afa70 con 0x7f64a41d7f10 2024-03-29T11:48:20.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.987+0000 7f64ac6a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:20.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.987+0000 7f64ac6a8640 10 monclient(hunting): authenticate will time out at 810.157747s 2024-03-29T11:48:20.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.987+0000 7f64a9c43640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f64a40b09f0 0x7f64a41d7800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:51538/0 (socket says 172.21.15.121:51538) 2024-03-29T11:48:20.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.987+0000 7f64a9c43640 1 -- 172.21.15.121:0/2240276578 learned_addr learned my addr 172.21.15.121:0/2240276578 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-29T11:48:20.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.987+0000 7f649affd640 10 client.?.objecter ms_handle_connect 0x7f64a40b09f0 2024-03-29T11:48:20.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.987+0000 7f649affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:20.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.987+0000 7f649affd640 10 client.?.objecter ms_handle_connect 0x7f64a409ac40 2024-03-29T11:48:20.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.987+0000 7f649affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:20.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.987+0000 7f649affd640 10 client.?.objecter ms_handle_connect 0x7f64a41d7f10 2024-03-29T11:48:20.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.987+0000 7f649affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:20.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.987+0000 7f649affd640 1 -- 172.21.15.121:0/2240276578 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1748694786 0 0) 0x7f64a41d72e0 con 0x7f64a40b09f0 2024-03-29T11:48:20.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.987+0000 7f649affd640 10 monclient(hunting): _negotiate 2024-03-29T11:48:20.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.987+0000 7f649affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:20.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.987+0000 7f649affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:20.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.987+0000 7f649affd640 10 monclient(hunting): my global_id is 4162 2024-03-29T11:48:20.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.987+0000 7f649affd640 1 -- 172.21.15.121:0/2240276578 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6488004eb0 con 0x7f64a40b09f0 2024-03-29T11:48:20.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.987+0000 7f649affd640 1 -- 172.21.15.121:0/2240276578 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2814807856 0 0) 0x7f64a40b19f0 con 0x7f64a409ac40 2024-03-29T11:48:20.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.987+0000 7f649affd640 10 monclient(hunting): _negotiate 2024-03-29T11:48:20.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.987+0000 7f649affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:20.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.987+0000 7f649affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:20.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.987+0000 7f649affd640 10 monclient(hunting): my global_id is 4155 2024-03-29T11:48:20.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.987+0000 7f649affd640 1 -- 172.21.15.121:0/2240276578 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64a41d72e0 con 0x7f64a409ac40 2024-03-29T11:48:20.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.987+0000 7f649affd640 1 -- 172.21.15.121:0/2240276578 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1368291314 0 0) 0x7f64a40afa70 con 0x7f64a41d7f10 2024-03-29T11:48:20.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.987+0000 7f649affd640 10 monclient(hunting): _negotiate 2024-03-29T11:48:20.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.987+0000 7f649affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:20.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.987+0000 7f649affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:20.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.987+0000 7f649affd640 10 monclient(hunting): my global_id is 4154 2024-03-29T11:48:20.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.987+0000 7f649affd640 1 -- 172.21.15.121:0/2240276578 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64a40b19f0 con 0x7f64a41d7f10 2024-03-29T11:48:20.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 1 -- 172.21.15.121:0/2240276578 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3226348064 0 0) 0x7f6488004eb0 con 0x7f64a40b09f0 2024-03-29T11:48:20.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 1 -- 172.21.15.121:0/2240276578 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64a40afa70 con 0x7f64a40b09f0 2024-03-29T11:48:20.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 1 -- 172.21.15.121:0/2240276578 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f6494001bc0 con 0x7f64a40b09f0 2024-03-29T11:48:20.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:20.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:48:20.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 10 monclient(hunting): dump: 2024-03-29T11:48:20.011 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:20.011 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:20.011 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:20.011 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:20.011 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:20.011 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:20.011 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:20.011 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:20.011 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:20.011 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:20.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:48:20.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 1 -- 172.21.15.121:0/2240276578 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1641676055 0 0) 0x7f64a40b19f0 con 0x7f64a41d7f10 2024-03-29T11:48:20.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 1 -- 172.21.15.121:0/2240276578 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6488004eb0 con 0x7f64a41d7f10 2024-03-29T11:48:20.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 1 -- 172.21.15.121:0/2240276578 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 333090902 0 0) 0x7f64a41d72e0 con 0x7f64a409ac40 2024-03-29T11:48:20.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 1 -- 172.21.15.121:0/2240276578 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64a40b19f0 con 0x7f64a409ac40 2024-03-29T11:48:20.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 1 -- 172.21.15.121:0/2240276578 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f649c0019f0 con 0x7f64a41d7f10 2024-03-29T11:48:20.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:20.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:48:20.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 10 monclient(hunting): dump: 2024-03-29T11:48:20.012 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:20.012 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:20.012 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:20.013 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:20.013 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:20.013 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:20.013 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:20.013 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:20.013 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:20.013 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:20.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 1 -- 172.21.15.121:0/2240276578 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f64a0001750 con 0x7f64a409ac40 2024-03-29T11:48:20.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:20.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:48:20.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 10 monclient(hunting): dump: 2024-03-29T11:48:20.013 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:20.013 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:20.014 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:20.014 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:20.014 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:20.014 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:20.014 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:20.014 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:20.014 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:20.014 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:20.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 1 -- 172.21.15.121:0/2240276578 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3550467887 0 0) 0x7f64a40afa70 con 0x7f64a40b09f0 2024-03-29T11:48:20.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 1 -- 172.21.15.121:0/2240276578 >> v1:172.21.15.98:6790/0 conn(0x7f64a41d7f10 legacy=0x7f64a4139080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:20.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 1 -- 172.21.15.121:0/2240276578 >> v1:172.21.15.98:6789/0 conn(0x7f64a409ac40 legacy=0x7f64a40aff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:20.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 10 monclient: _finish_hunting 0 2024-03-29T11:48:20.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 1 monclient: found mon.b 2024-03-29T11:48:20.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:20.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:20.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 1 -- 172.21.15.121:0/2240276578 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a40acf70 con 0x7f64a40b09f0 2024-03-29T11:48:20.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 10 monclient: _finish_auth 0 2024-03-29T11:48:20.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 10 monclient: _check_auth_tickets 2024-03-29T11:48:20.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:20.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f64ac6a8640 5 monclient: authenticate success, global_id 4162 2024-03-29T11:48:20.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f64ac6a8640 10 monclient: _renew_subs 2024-03-29T11:48:20.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f64ac6a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:20.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f64ac6a8640 1 -- 172.21.15.121:0/2240276578 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64a40b0680 con 0x7f64a40b09f0 2024-03-29T11:48:20.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f64ac6a8640 10 client.4162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:48:20.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f64ac6a8640 10 monclient: _renew_subs 2024-03-29T11:48:20.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f64ac6a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:20.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f64ac6a8640 1 -- 172.21.15.121:0/2240276578 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f64a4139970 con 0x7f64a40b09f0 2024-03-29T11:48:20.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f64ac6a8640 1 librados: init done 2024-03-29T11:48:20.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 1 -- 172.21.15.121:0/2240276578 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6494004bf0 con 0x7f64a40b09f0 2024-03-29T11:48:20.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:20.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 1 -- 172.21.15.121:0/2240276578 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f6494004f00 con 0x7f64a40b09f0 2024-03-29T11:48:20.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:20.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:48:20.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f64a8c41640 4 set_mon_vals no callback set 2024-03-29T11:48:20.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 10 monclient: dump: 2024-03-29T11:48:20.016 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:48:20.017 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:20.017 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:20.017 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:20.017 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:20.017 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:48:20.017 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:20.017 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:20.017 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:20.017 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:48:20.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 1 -- 172.21.15.121:0/2240276578 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f64940053f0 con 0x7f64a40b09f0 2024-03-29T11:48:20.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 1 -- 172.21.15.121:0/2240276578 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 2583741553 0 0) 0x7f6494006be0 con 0x7f64a40b09f0 2024-03-29T11:48:20.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 10 client.4162.objecter ms_dispatch 0x7f64a4000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T11:48:20.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 3 client.4162.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T11:48:20.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 3 client.4162.objecter handle_osd_map decoding full epoch 9 2024-03-29T11:48:20.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.988+0000 7f649affd640 20 client.4162.objecter dump_active .. 0 homeless 2024-03-29T11:48:20.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.989+0000 7f64ac6a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:20.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.989+0000 7f64ac6a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:20.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.989+0000 7f64ac6a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:20.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.989+0000 7f64ac6a8640 1 -- 172.21.15.121:0/2240276578 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f648c005080 con 0x7f64a40b09f0 2024-03-29T11:48:20.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.991+0000 7f649affd640 1 -- 172.21.15.121:0/2240276578 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6494006630 con 0x7f64a40b09f0 2024-03-29T11:48:20.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.991+0000 7f649affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:20.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:19.991+0000 7f649affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:48:20.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:20.063+0000 7f64ac6a8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "48b0600d-02cc-4677-9176-be8248744f5e", "id": 7}] 2024-03-29T11:48:20.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:20.063+0000 7f64ac6a8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "48b0600d-02cc-4677-9176-be8248744f5e", "id": 7}] 2024-03-29T11:48:20.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:20.063+0000 7f64ac6a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:20.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:20.063+0000 7f64ac6a8640 1 -- 172.21.15.121:0/2240276578 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd new", "uuid": "48b0600d-02cc-4677-9176-be8248744f5e", "id": 7} v 0) v1 -- 0x7f648c0058b0 con 0x7f64a40b09f0 2024-03-29T11:48:20.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:20.066+0000 7f649affd640 1 -- 172.21.15.121:0/2240276578 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "48b0600d-02cc-4677-9176-be8248744f5e", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 1371151107 0 1371665406) 0x7f64940068a0 con 0x7f64a40b09f0 2024-03-29T11:48:20.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:20.066+0000 7f649affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "48b0600d-02cc-4677-9176-be8248744f5e", "id": 7}] 2024-03-29T11:48:20.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:20.066+0000 7f649affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:48:20.072 INFO:teuthology.orchestra.run.smithi121.stdout:7 2024-03-29T11:48:20.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:20.067+0000 7f64ac6a8640 10 librados: watch_flush enter 2024-03-29T11:48:20.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:20.067+0000 7f64ac6a8640 10 librados: watch_flush exit 2024-03-29T11:48:20.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:20.067+0000 7f64ac6a8640 20 client.4162.objecter shutdown clearing up homeless session... 2024-03-29T11:48:20.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:20.067+0000 7f64ac6a8640 10 client.4162.objecter successfully canceled tick 2024-03-29T11:48:20.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:20.067+0000 7f64ac6a8640 10 monclient: shutdown 2024-03-29T11:48:20.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:20.067+0000 7f64ac6a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:20.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:20.067+0000 7f64ac6a8640 1 -- 172.21.15.121:0/2240276578 >> v1:172.21.15.121:6789/0 conn(0x7f64a40b09f0 legacy=0x7f64a41d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:20.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:20.067+0000 7f64ac6a8640 1 -- 172.21.15.121:0/2240276578 shutdown_connections 2024-03-29T11:48:20.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:20.067+0000 7f64ac6a8640 1 -- 172.21.15.121:0/2240276578 >> 172.21.15.121:0/2240276578 conn(0x7f64a4066b00 msgr2=0x7f64a40b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:20.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:20.067+0000 7f64ac6a8640 1 -- 172.21.15.121:0/2240276578 shutdown_connections 2024-03-29T11:48:20.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:20.067+0000 7f64ac6a8640 1 -- 172.21.15.121:0/2240276578 wait complete. 2024-03-29T11:48:20.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:48:20.068+0000 7f64ac6a8640 1 librados: shutdown 2024-03-29T11:48:20.083 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T11:48:20.083 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T11:48:20.086 INFO:tasks.ceph.osd.0:Started 2024-03-29T11:48:20.086 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T11:48:20.086 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T11:48:20.088 INFO:tasks.ceph.osd.1:Started 2024-03-29T11:48:20.088 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T11:48:20.088 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T11:48:20.090 INFO:tasks.ceph.osd.2:Started 2024-03-29T11:48:20.090 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T11:48:20.090 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T11:48:20.090 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:48:20.084+0000 7f4e33c10040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:48:20.092 INFO:tasks.ceph.osd.3:Started 2024-03-29T11:48:20.092 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T11:48:20.092 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T11:48:20.094 INFO:tasks.ceph.osd.4:Started 2024-03-29T11:48:20.094 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T11:48:20.094 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T11:48:20.096 INFO:tasks.ceph.osd.5:Started 2024-03-29T11:48:20.096 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T11:48:20.096 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T11:48:20.098 INFO:tasks.ceph.osd.6:Started 2024-03-29T11:48:20.098 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T11:48:20.099 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T11:48:20.100 INFO:tasks.ceph.osd.7:Started 2024-03-29T11:48:20.101 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:48:20.104 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T11:48:20.098+0000 7faaea042040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:48:20.159 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T11:48:20.154+0000 7faaea042040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:48:20.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.223+0000 7f30abc3e640 10 monclient: get_monmap_and_config 2024-03-29T11:48:20.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.223+0000 7f30abc3e640 10 monclient: build_initial_monmap 2024-03-29T11:48:20.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.223+0000 7f30abc3e640 10 monclient: monmap: 2024-03-29T11:48:20.228 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-03-29T11:48:20.229 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:20.229 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-29T11:48:20.223992+0000 2024-03-29T11:48:20.229 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-29T11:48:20.223992+0000 2024-03-29T11:48:20.229 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:20.229 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:20.229 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:48:20.229 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:48:20.229 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:48:20.229 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-29T11:48:20.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.223+0000 7f30abc3e640 1 Processor -- start 2024-03-29T11:48:20.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.223+0000 7f30abc3e640 1 -- start start 2024-03-29T11:48:20.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.223+0000 7f30abc3e640 10 monclient: init 2024-03-29T11:48:20.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.224+0000 7f30abc3e640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:20.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.224+0000 7f30abc3e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:48:20.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.224+0000 7f30abc3e640 10 monclient(hunting): picked mon.noname-b con 0x7f30a40b8240 addr v1:172.21.15.98:6790/0 2024-03-29T11:48:20.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.224+0000 7f30abc3e640 10 monclient(hunting): picked mon.noname-c con 0x7f30a40b1f60 addr v1:172.21.15.121:6789/0 2024-03-29T11:48:20.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.224+0000 7f30abc3e640 10 monclient(hunting): picked mon.noname-a con 0x7f30a409c8c0 addr v1:172.21.15.98:6789/0 2024-03-29T11:48:20.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.224+0000 7f30abc3e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a409bd80 con 0x7f30a409c8c0 2024-03-29T11:48:20.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.224+0000 7f30abc3e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a409b960 con 0x7f30a40b1f60 2024-03-29T11:48:20.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.224+0000 7f30abc3e640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a40b2ec0 con 0x7f30a40b8240 2024-03-29T11:48:20.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.224+0000 7f30abc3e640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:20.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.224+0000 7f30abc3e640 10 monclient(hunting): authenticate will time out at 807.309496s 2024-03-29T11:48:20.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.224+0000 7f30aa1db640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f30a409c8c0 0x7f30a414d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:60904/0 (socket says 172.21.15.98:60904) 2024-03-29T11:48:20.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.224+0000 7f30aa1db640 1 -- 172.21.15.98:0/2126099354 learned_addr learned my addr 172.21.15.98:0/2126099354 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:48:20.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.224+0000 7f30a99da640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f30a40b8240 0x7f30a413f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:49634/0 (socket says 172.21.15.98:49634) 2024-03-29T11:48:20.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.224+0000 7f30a89d8640 1 -- 172.21.15.98:0/2126099354 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3931726911 0 0) 0x7f30a40b2ec0 con 0x7f30a40b8240 2024-03-29T11:48:20.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.224+0000 7f30a89d8640 10 monclient(hunting): _negotiate 2024-03-29T11:48:20.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.224+0000 7f30a89d8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:20.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.224+0000 7f30a89d8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:20.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.224+0000 7f30a89d8640 10 monclient(hunting): my global_id is 4181 2024-03-29T11:48:20.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.224+0000 7f30a89d8640 1 -- 172.21.15.98:0/2126099354 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3090003520 con 0x7f30a40b8240 2024-03-29T11:48:20.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.224+0000 7f30a89d8640 1 -- 172.21.15.98:0/2126099354 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3626471792 0 0) 0x7f30a409bd80 con 0x7f30a409c8c0 2024-03-29T11:48:20.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.224+0000 7f30a89d8640 10 monclient(hunting): _negotiate 2024-03-29T11:48:20.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.224+0000 7f30a89d8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:20.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.224+0000 7f30a89d8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:20.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.224+0000 7f30a89d8640 10 monclient(hunting): my global_id is 4182 2024-03-29T11:48:20.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.224+0000 7f30a89d8640 1 -- 172.21.15.98:0/2126099354 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30a40b2ec0 con 0x7f30a409c8c0 2024-03-29T11:48:20.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.224+0000 7f30a89d8640 1 -- 172.21.15.98:0/2126099354 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1039089568 0 0) 0x7f30a409b960 con 0x7f30a40b1f60 2024-03-29T11:48:20.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.225+0000 7f30a89d8640 10 monclient(hunting): _negotiate 2024-03-29T11:48:20.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.225+0000 7f30a89d8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:20.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.225+0000 7f30a89d8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:20.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.225+0000 7f30a89d8640 10 monclient(hunting): my global_id is 4189 2024-03-29T11:48:20.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.225+0000 7f30a89d8640 1 -- 172.21.15.98:0/2126099354 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30a409bd80 con 0x7f30a40b1f60 2024-03-29T11:48:20.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.225+0000 7f30a89d8640 1 -- 172.21.15.98:0/2126099354 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2057911352 0 0) 0x7f3090003520 con 0x7f30a40b8240 2024-03-29T11:48:20.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.225+0000 7f30a89d8640 1 -- 172.21.15.98:0/2126099354 >> v1:172.21.15.121:6789/0 conn(0x7f30a40b1f60 legacy=0x7f30a409c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:20.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.225+0000 7f30a89d8640 1 -- 172.21.15.98:0/2126099354 >> v1:172.21.15.98:6789/0 conn(0x7f30a409c8c0 legacy=0x7f30a414d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:20.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.225+0000 7f30a89d8640 10 monclient: _finish_hunting 0 2024-03-29T11:48:20.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.225+0000 7f30a89d8640 1 monclient: found mon.noname-b 2024-03-29T11:48:20.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.225+0000 7f30a89d8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:20.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.225+0000 7f30a89d8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:48:20.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.225+0000 7f30a89d8640 1 -- 172.21.15.98:0/2126099354 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a413fb60 con 0x7f30a40b8240 2024-03-29T11:48:20.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.225+0000 7f30a89d8640 10 monclient: _finish_auth 0 2024-03-29T11:48:20.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.225+0000 7f30a89d8640 10 monclient: _check_auth_tickets 2024-03-29T11:48:20.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.225+0000 7f30a89d8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:20.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.225+0000 7f30a89d8640 1 -- 172.21.15.98:0/2126099354 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f3094001690 con 0x7f30a40b8240 2024-03-29T11:48:20.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.225+0000 7f30abc3e640 5 monclient: authenticate success, global_id 4181 2024-03-29T11:48:20.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.225+0000 7f30abc3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:48:20.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.225+0000 7f30a89d8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:20.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.225+0000 7f30a89d8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:48:20.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.225+0000 7f30a89d8640 10 monclient: dump: 2024-03-29T11:48:20.242 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-29T11:48:20.242 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:20.242 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:20.242 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:20.242 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:20.242 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:20.242 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:20.242 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:20.242 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:20.242 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-29T11:48:20.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.225+0000 7f30abc3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:48:20.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.225+0000 7f30a89d8640 1 -- 172.21.15.98:0/2126099354 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3094001870 con 0x7f30a40b8240 2024-03-29T11:48:20.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.225+0000 7f30a89d8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:20.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.225+0000 7f30a89d8640 1 -- 172.21.15.98:0/2126099354 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f3094001c80 con 0x7f30a40b8240 2024-03-29T11:48:20.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.225+0000 7f30abc3e640 10 monclient: get_monmap_and_config success 2024-03-29T11:48:20.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.225+0000 7f30abc3e640 4 set_mon_vals no callback set 2024-03-29T11:48:20.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.225+0000 7f30a89d8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:20.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.225+0000 7f30a89d8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:48:20.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.225+0000 7f30a89d8640 10 monclient: dump: 2024-03-29T11:48:20.243 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-29T11:48:20.243 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:20.243 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:20.243 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:20.243 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:20.243 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:20.243 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:20.243 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:20.243 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:20.243 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-29T11:48:20.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.225+0000 7f30abc3e640 10 monclient: shutdown 2024-03-29T11:48:20.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.225+0000 7f30abc3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:20.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.225+0000 7f30abc3e640 1 -- 172.21.15.98:0/2126099354 >> v1:172.21.15.98:6790/0 conn(0x7f30a40b8240 legacy=0x7f30a413f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:20.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.225+0000 7f30abc3e640 1 -- 172.21.15.98:0/2126099354 shutdown_connections 2024-03-29T11:48:20.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.225+0000 7f30abc3e640 1 -- 172.21.15.98:0/2126099354 >> 172.21.15.98:0/2126099354 conn(0x7f30a4066b00 msgr2=0x7f30a4066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:20.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.226+0000 7f30abc3e640 1 -- 172.21.15.98:0/2126099354 shutdown_connections 2024-03-29T11:48:20.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.226+0000 7f30abc3e640 1 -- 172.21.15.98:0/2126099354 wait complete. 2024-03-29T11:48:20.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.226+0000 7f30abc3e640 10 monclient: build_initial_monmap 2024-03-29T11:48:20.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.226+0000 7f30abc3e640 10 monclient: monmap: 2024-03-29T11:48:20.244 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-03-29T11:48:20.244 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:48:20.244 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-03-29T11:48:20.244 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-03-29T11:48:20.244 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:20.245 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:20.245 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:48:20.245 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:48:20.245 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:48:20.245 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-29T11:48:20.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.226+0000 7f30abc3e640 1 librados: starting msgr at 2024-03-29T11:48:20.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.226+0000 7f30abc3e640 1 librados: starting objecter 2024-03-29T11:48:20.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.227+0000 7f30abc3e640 1 Processor -- start 2024-03-29T11:48:20.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.227+0000 7f30abc3e640 1 -- start start 2024-03-29T11:48:20.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.227+0000 7f30abc3e640 1 librados: setting wanted keys 2024-03-29T11:48:20.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.227+0000 7f30abc3e640 1 librados: calling monclient init 2024-03-29T11:48:20.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.227+0000 7f30abc3e640 10 monclient: init 2024-03-29T11:48:20.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.227+0000 7f30abc3e640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:20.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.227+0000 7f30abc3e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:48:20.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.227+0000 7f30abc3e640 10 monclient(hunting): picked mon.noname-c con 0x7f30a409c8c0 addr v1:172.21.15.98:6790/0 2024-03-29T11:48:20.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.227+0000 7f30abc3e640 10 monclient(hunting): picked mon.noname-a con 0x7f30a40b8240 addr v1:172.21.15.98:6789/0 2024-03-29T11:48:20.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.227+0000 7f30abc3e640 10 monclient(hunting): picked mon.noname-b con 0x7f30a41d4c20 addr v1:172.21.15.121:6789/0 2024-03-29T11:48:20.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.227+0000 7f30a91d9640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f30a40b8240 0x7f30a41d4510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:60916/0 (socket says 172.21.15.98:60916) 2024-03-29T11:48:20.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.227+0000 7f30a99da640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f30a409c8c0 0x7f30a41327b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:49640/0 (socket says 172.21.15.98:49640) 2024-03-29T11:48:20.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.227+0000 7f30a91d9640 1 -- 172.21.15.98:0/1855834053 learned_addr learned my addr 172.21.15.98:0/1855834053 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:48:20.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.227+0000 7f30abc3e640 1 -- 172.21.15.98:0/1855834053 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a41d4080 con 0x7f30a40b8240 2024-03-29T11:48:20.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.227+0000 7f309a7fc640 10 client.?.objecter ms_handle_connect 0x7f30a409c8c0 2024-03-29T11:48:20.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.227+0000 7f309a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:20.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.227+0000 7f309a7fc640 10 client.?.objecter ms_handle_connect 0x7f30a40b8240 2024-03-29T11:48:20.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.227+0000 7f309a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:20.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.227+0000 7f30abc3e640 1 -- 172.21.15.98:0/1855834053 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a41345e0 con 0x7f30a41d4c20 2024-03-29T11:48:20.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.227+0000 7f30abc3e640 1 -- 172.21.15.98:0/1855834053 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a4134770 con 0x7f30a409c8c0 2024-03-29T11:48:20.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.227+0000 7f309a7fc640 10 client.?.objecter ms_handle_connect 0x7f30a41d4c20 2024-03-29T11:48:20.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.227+0000 7f309a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:20.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.227+0000 7f30abc3e640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:20.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.227+0000 7f30abc3e640 10 monclient(hunting): authenticate will time out at 807.313004s 2024-03-29T11:48:20.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.227+0000 7f309a7fc640 1 -- 172.21.15.98:0/1855834053 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2807278710 0 0) 0x7f30a41d4080 con 0x7f30a40b8240 2024-03-29T11:48:20.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:48:20.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:20.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:20.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 10 monclient(hunting): my global_id is 4185 2024-03-29T11:48:20.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 1 -- 172.21.15.98:0/1855834053 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3088004eb0 con 0x7f30a40b8240 2024-03-29T11:48:20.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 1 -- 172.21.15.98:0/1855834053 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1222800946 0 0) 0x7f30a41345e0 con 0x7f30a41d4c20 2024-03-29T11:48:20.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:48:20.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:20.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:20.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 10 monclient(hunting): my global_id is 4192 2024-03-29T11:48:20.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 1 -- 172.21.15.98:0/1855834053 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30a41d4080 con 0x7f30a41d4c20 2024-03-29T11:48:20.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 1 -- 172.21.15.98:0/1855834053 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1364812712 0 0) 0x7f30a4134770 con 0x7f30a409c8c0 2024-03-29T11:48:20.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:48:20.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:20.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:20.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 10 monclient(hunting): my global_id is 4184 2024-03-29T11:48:20.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 1 -- 172.21.15.98:0/1855834053 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30a41345e0 con 0x7f30a409c8c0 2024-03-29T11:48:20.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 1 -- 172.21.15.98:0/1855834053 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 401510380 0 0) 0x7f3088004eb0 con 0x7f30a40b8240 2024-03-29T11:48:20.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 1 -- 172.21.15.98:0/1855834053 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30a4134770 con 0x7f30a40b8240 2024-03-29T11:48:20.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 1 -- 172.21.15.98:0/1855834053 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f30a0001950 con 0x7f30a40b8240 2024-03-29T11:48:20.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:20.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:48:20.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 10 monclient(hunting): dump: 2024-03-29T11:48:20.248 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-29T11:48:20.249 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:20.249 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:20.249 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:20.249 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:20.249 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:20.249 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:20.249 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:20.249 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:20.249 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-29T11:48:20.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:48:20.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 1 -- 172.21.15.98:0/1855834053 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 996044348 0 0) 0x7f30a41d4080 con 0x7f30a41d4c20 2024-03-29T11:48:20.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 1 -- 172.21.15.98:0/1855834053 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3088004eb0 con 0x7f30a41d4c20 2024-03-29T11:48:20.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 1 -- 172.21.15.98:0/1855834053 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f309c0015d0 con 0x7f30a41d4c20 2024-03-29T11:48:20.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:20.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:48:20.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 10 monclient(hunting): dump: 2024-03-29T11:48:20.250 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-29T11:48:20.250 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:20.250 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:20.250 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:20.250 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:20.250 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:20.250 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:20.250 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:20.250 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:20.250 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-29T11:48:20.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 1 -- 172.21.15.98:0/1855834053 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3270880991 0 0) 0x7f30a41345e0 con 0x7f30a409c8c0 2024-03-29T11:48:20.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 1 -- 172.21.15.98:0/1855834053 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30a41d4080 con 0x7f30a409c8c0 2024-03-29T11:48:20.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 1 -- 172.21.15.98:0/1855834053 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f3094001b80 con 0x7f30a409c8c0 2024-03-29T11:48:20.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:20.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:48:20.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 10 monclient(hunting): dump: 2024-03-29T11:48:20.251 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-29T11:48:20.251 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:20.251 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:20.251 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:20.251 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:20.251 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:20.251 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:20.251 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:20.251 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:20.251 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-29T11:48:20.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 1 -- 172.21.15.98:0/1855834053 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 669047377 0 0) 0x7f3088004eb0 con 0x7f30a41d4c20 2024-03-29T11:48:20.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 1 -- 172.21.15.98:0/1855834053 >> v1:172.21.15.98:6790/0 conn(0x7f30a409c8c0 legacy=0x7f30a41327b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:20.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 1 -- 172.21.15.98:0/1855834053 >> v1:172.21.15.98:6789/0 conn(0x7f30a40b8240 legacy=0x7f30a41d4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:20.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:48:20.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 1 monclient: found mon.b 2024-03-29T11:48:20.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:20.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:20.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 1 -- 172.21.15.98:0/1855834053 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a4132ec0 con 0x7f30a41d4c20 2024-03-29T11:48:20.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:48:20.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:48:20.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:20.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f30abc3e640 5 monclient: authenticate success, global_id 4192 2024-03-29T11:48:20.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f30abc3e640 10 monclient: _renew_subs 2024-03-29T11:48:20.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 1 -- 172.21.15.98:0/1855834053 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f309c002030 con 0x7f30a41d4c20 2024-03-29T11:48:20.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f30abc3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:20.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f30abc3e640 1 -- 172.21.15.98:0/1855834053 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30a4131aa0 con 0x7f30a41d4c20 2024-03-29T11:48:20.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:20.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 1 -- 172.21.15.98:0/1855834053 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f309c004b60 con 0x7f30a41d4c20 2024-03-29T11:48:20.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:20.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:48:20.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 10 monclient: dump: 2024-03-29T11:48:20.253 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-29T11:48:20.253 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:20.253 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:20.253 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:20.253 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:20.253 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:20.253 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:20.253 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:20.253 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:20.253 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-29T11:48:20.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309bfff640 4 set_mon_vals no callback set 2024-03-29T11:48:20.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f30abc3e640 10 client.4192.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:48:20.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f30abc3e640 10 monclient: _renew_subs 2024-03-29T11:48:20.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f30abc3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:20.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f30abc3e640 1 -- 172.21.15.98:0/1855834053 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f30a41e33d0 con 0x7f30a41d4c20 2024-03-29T11:48:20.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f30abc3e640 1 librados: init done 2024-03-29T11:48:20.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 1 -- 172.21.15.98:0/1855834053 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f309c0050d0 con 0x7f30a41d4c20 2024-03-29T11:48:20.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 1 -- 172.21.15.98:0/1855834053 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f309c005a50 con 0x7f30a41d4c20 2024-03-29T11:48:20.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 10 client.4192.objecter ms_dispatch 0x7f30a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:48:20.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 3 client.4192.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:48:20.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 3 client.4192.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:48:20.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.228+0000 7f309a7fc640 20 client.4192.objecter dump_active .. 0 homeless 2024-03-29T11:48:20.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.229+0000 7f30abc3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:20.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.229+0000 7f30abc3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:20.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.230+0000 7f30abc3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:20.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.230+0000 7f30abc3e640 1 -- 172.21.15.98:0/1855834053 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30a409b170 con 0x7f30a41d4c20 2024-03-29T11:48:20.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.232+0000 7f309a7fc640 1 -- 172.21.15.98:0/1855834053 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f309c005d70 con 0x7f30a41d4c20 2024-03-29T11:48:20.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.232+0000 7f309a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:20.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.232+0000 7f309a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:48:20.255 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:48:20.233+0000 7f4e33c10040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:48:20.285 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:48:20.279+0000 7f4e33c10040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:48:20.308 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T11:48:20.303+0000 7faaea042040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:48:20.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.321+0000 7f30abc3e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:20.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.322+0000 7f30abc3e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:20.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.322+0000 7f30abc3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:20.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.322+0000 7f30abc3e640 1 -- 172.21.15.98:0/1855834053 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30a41d3eb0 con 0x7f30a41d4c20 2024-03-29T11:48:20.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.322+0000 7f309a7fc640 1 -- 172.21.15.98:0/1855834053 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f309c0054b0 con 0x7f30a41d4c20 2024-03-29T11:48:20.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.322+0000 7f309a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:20.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.322+0000 7f309a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:48:20.328 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T11:48:20.328 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:48:20.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.323+0000 7f30abc3e640 10 librados: watch_flush enter 2024-03-29T11:48:20.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.323+0000 7f30abc3e640 10 librados: watch_flush exit 2024-03-29T11:48:20.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.323+0000 7f30abc3e640 20 client.4192.objecter shutdown clearing up homeless session... 2024-03-29T11:48:20.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.323+0000 7f30abc3e640 10 client.4192.objecter successfully canceled tick 2024-03-29T11:48:20.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.323+0000 7f30abc3e640 10 monclient: shutdown 2024-03-29T11:48:20.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.323+0000 7f30abc3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:20.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.323+0000 7f30abc3e640 1 -- 172.21.15.98:0/1855834053 >> v1:172.21.15.121:6789/0 conn(0x7f30a41d4c20 legacy=0x7f30a41e1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:20.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.323+0000 7f30abc3e640 1 -- 172.21.15.98:0/1855834053 shutdown_connections 2024-03-29T11:48:20.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.323+0000 7f30abc3e640 1 -- 172.21.15.98:0/1855834053 >> 172.21.15.98:0/1855834053 conn(0x7f30a4066b00 msgr2=0x7f30a4069bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:20.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.323+0000 7f30abc3e640 1 -- 172.21.15.98:0/1855834053 shutdown_connections 2024-03-29T11:48:20.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.323+0000 7f30abc3e640 1 -- 172.21.15.98:0/1855834053 wait complete. 2024-03-29T11:48:20.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T11:48:20.323+0000 7f30abc3e640 1 librados: shutdown 2024-03-29T11:48:20.339 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T11:48:20.339 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T11:48:20.395 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:48:20.389+0000 7f4e33c10040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:48:20.405 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T11:48:20.399+0000 7faaea042040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:48:20.491 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:48:20.485+0000 7f4e33c10040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:48:20.559 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T11:48:20.553+0000 7faaea042040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:48:20.600 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T11:48:20.595+0000 7faaea042040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:48:20.642 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:48:20.636+0000 7f4e33c10040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:48:20.684 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:48:20.678+0000 7f4e33c10040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:48:20.690 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-03-29T11:48:20.684+0000 7f1ed1a37640 -1 Falling back to public interface 2024-03-29T11:48:20.698 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-03-29T11:48:20.692+0000 7fc352bfa640 -1 Falling back to public interface 2024-03-29T11:48:20.698 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-03-29T11:48:20.692+0000 7f57dcb9a640 -1 Falling back to public interface 2024-03-29T11:48:20.700 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-03-29T11:48:20.694+0000 7fb0655bd640 -1 Falling back to public interface 2024-03-29T11:48:20.700 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-03-29T11:48:20.695+0000 7f4f9216b640 -1 Falling back to public interface 2024-03-29T11:48:20.711 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-03-29T11:48:20.706+0000 7f446d217640 -1 Falling back to public interface 2024-03-29T11:48:20.712 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-03-29T11:48:20.707+0000 7f96c4611640 -1 Falling back to public interface 2024-03-29T11:48:20.719 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-03-29T11:48:20.714+0000 7ff874e11640 -1 Falling back to public interface 2024-03-29T11:48:20.755 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T11:48:20.750+0000 7faaea042040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:48:20.843 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:48:20.837+0000 7f4e33c10040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:48:21.144 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:48:21.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.228+0000 7f3175686640 10 monclient: get_monmap_and_config 2024-03-29T11:48:21.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.228+0000 7f3175686640 10 monclient: build_initial_monmap 2024-03-29T11:48:21.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.228+0000 7f3175686640 10 monclient: monmap: 2024-03-29T11:48:21.234 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:48:21.234 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:21.235 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:48:21.229861+0000 2024-03-29T11:48:21.235 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:48:21.229861+0000 2024-03-29T11:48:21.235 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:21.235 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:21.235 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:48:21.235 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:48:21.235 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:48:21.235 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:21.243 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.229+0000 7f3175686640 1 Processor -- start 2024-03-29T11:48:21.243 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.229+0000 7f3175686640 1 -- start start 2024-03-29T11:48:21.243 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.229+0000 7f3175686640 10 monclient: init 2024-03-29T11:48:21.243 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f3175686640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:21.243 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f3175686640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:48:21.243 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f3175686640 10 monclient(hunting): picked mon.noname-a con 0x7f31701422b0 addr v1:172.21.15.98:6789/0 2024-03-29T11:48:21.243 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f3175686640 10 monclient(hunting): picked mon.noname-b con 0x7f3170142690 addr v1:172.21.15.98:6790/0 2024-03-29T11:48:21.243 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f3175686640 10 monclient(hunting): picked mon.noname-c con 0x7f31700b17d0 addr v1:172.21.15.121:6789/0 2024-03-29T11:48:21.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f3175686640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3170142c80 con 0x7f31701422b0 2024-03-29T11:48:21.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f3175686640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31700aae40 con 0x7f31700b17d0 2024-03-29T11:48:21.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f3175686640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f317013fbd0 con 0x7f3170142690 2024-03-29T11:48:21.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f3175686640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:21.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f3175686640 10 monclient(hunting): authenticate will time out at 808.315201s 2024-03-29T11:48:21.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316effd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f31701422b0 0x7f317013f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:60930/0 (socket says 172.21.15.98:60930) 2024-03-29T11:48:21.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316e7fc640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f3170142690 0x7f31700b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:49646/0 (socket says 172.21.15.98:49646) 2024-03-29T11:48:21.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316effd640 1 -- 172.21.15.98:0/3470174891 learned_addr learned my addr 172.21.15.98:0/3470174891 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:48:21.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316dffb640 1 -- 172.21.15.98:0/3470174891 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 668591833 0 0) 0x7f3170142c80 con 0x7f31701422b0 2024-03-29T11:48:21.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:48:21.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:21.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:21.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316dffb640 10 monclient(hunting): my global_id is 4188 2024-03-29T11:48:21.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316dffb640 1 -- 172.21.15.98:0/3470174891 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3158003520 con 0x7f31701422b0 2024-03-29T11:48:21.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316dffb640 1 -- 172.21.15.98:0/3470174891 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2673760572 0 0) 0x7f317013fbd0 con 0x7f3170142690 2024-03-29T11:48:21.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:48:21.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:21.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:21.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316dffb640 10 monclient(hunting): my global_id is 4187 2024-03-29T11:48:21.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316dffb640 1 -- 172.21.15.98:0/3470174891 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3170142c80 con 0x7f3170142690 2024-03-29T11:48:21.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316dffb640 1 -- 172.21.15.98:0/3470174891 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3690696749 0 0) 0x7f31700aae40 con 0x7f31700b17d0 2024-03-29T11:48:21.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:48:21.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:21.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:21.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316dffb640 10 monclient(hunting): my global_id is 4195 2024-03-29T11:48:21.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316dffb640 1 -- 172.21.15.98:0/3470174891 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f317013fbd0 con 0x7f31700b17d0 2024-03-29T11:48:21.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316dffb640 1 -- 172.21.15.98:0/3470174891 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1649350077 0 0) 0x7f3170142c80 con 0x7f3170142690 2024-03-29T11:48:21.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316dffb640 1 -- 172.21.15.98:0/3470174891 >> v1:172.21.15.121:6789/0 conn(0x7f31700b17d0 legacy=0x7f31700b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:21.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316dffb640 1 -- 172.21.15.98:0/3470174891 >> v1:172.21.15.98:6789/0 conn(0x7f31701422b0 legacy=0x7f317013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:21.246 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:48:21.246 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316dffb640 1 monclient: found mon.noname-b 2024-03-29T11:48:21.246 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:21.246 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:48:21.246 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316dffb640 1 -- 172.21.15.98:0/3470174891 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3170151940 con 0x7f3170142690 2024-03-29T11:48:21.246 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316dffb640 10 monclient: _finish_auth 0 2024-03-29T11:48:21.246 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:48:21.246 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:21.246 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316dffb640 1 -- 172.21.15.98:0/3470174891 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f31640019f0 con 0x7f3170142690 2024-03-29T11:48:21.246 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f3175686640 5 monclient: authenticate success, global_id 4187 2024-03-29T11:48:21.246 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f3175686640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:48:21.246 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:21.246 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:48:21.246 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316dffb640 10 monclient: dump: 2024-03-29T11:48:21.247 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:48:21.247 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:21.247 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:21.247 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:21.247 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:21.247 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:21.247 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:21.247 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:21.247 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:21.247 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:21.247 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f3175686640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:48:21.247 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316dffb640 1 -- 172.21.15.98:0/3470174891 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3164001b50 con 0x7f3170142690 2024-03-29T11:48:21.247 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:21.247 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316dffb640 1 -- 172.21.15.98:0/3470174891 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f3164001f20 con 0x7f3170142690 2024-03-29T11:48:21.247 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f3175686640 10 monclient: get_monmap_and_config success 2024-03-29T11:48:21.248 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:21.248 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f3175686640 4 set_mon_vals no callback set 2024-03-29T11:48:21.248 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:48:21.248 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.230+0000 7f316dffb640 10 monclient: dump: 2024-03-29T11:48:21.248 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:48:21.248 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:21.248 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:21.248 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:21.248 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:21.248 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:21.248 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:21.248 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:21.248 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:21.248 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:21.248 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.231+0000 7f3175686640 10 monclient: shutdown 2024-03-29T11:48:21.249 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.231+0000 7f3175686640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:21.249 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.231+0000 7f3175686640 1 -- 172.21.15.98:0/3470174891 >> v1:172.21.15.98:6790/0 conn(0x7f3170142690 legacy=0x7f31700b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:21.249 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.231+0000 7f3175686640 1 -- 172.21.15.98:0/3470174891 shutdown_connections 2024-03-29T11:48:21.249 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.231+0000 7f3175686640 1 -- 172.21.15.98:0/3470174891 >> 172.21.15.98:0/3470174891 conn(0x7f3170066a00 msgr2=0x7f3170066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:21.249 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.231+0000 7f3175686640 1 -- 172.21.15.98:0/3470174891 shutdown_connections 2024-03-29T11:48:21.249 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.231+0000 7f3175686640 1 -- 172.21.15.98:0/3470174891 wait complete. 2024-03-29T11:48:21.249 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.232+0000 7f3175686640 10 monclient: build_initial_monmap 2024-03-29T11:48:21.249 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.232+0000 7f3175686640 10 monclient: monmap: 2024-03-29T11:48:21.249 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:48:21.249 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:48:21.249 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:48:21.249 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:48:21.249 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:21.249 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:21.249 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:48:21.250 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:48:21.250 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:48:21.250 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:21.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.232+0000 7f3175686640 1 librados: starting msgr at 2024-03-29T11:48:21.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.232+0000 7f3175686640 1 librados: starting objecter 2024-03-29T11:48:21.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.232+0000 7f3175686640 1 Processor -- start 2024-03-29T11:48:21.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.232+0000 7f3175686640 1 -- start start 2024-03-29T11:48:21.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.232+0000 7f3175686640 1 librados: setting wanted keys 2024-03-29T11:48:21.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.232+0000 7f3175686640 1 librados: calling monclient init 2024-03-29T11:48:21.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.232+0000 7f3175686640 10 monclient: init 2024-03-29T11:48:21.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f3175686640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:21.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f3175686640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:48:21.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f3175686640 10 monclient(hunting): picked mon.noname-b con 0x7f31700b17d0 addr v1:172.21.15.121:6789/0 2024-03-29T11:48:21.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f3175686640 10 monclient(hunting): picked mon.noname-a con 0x7f31701422b0 addr v1:172.21.15.98:6789/0 2024-03-29T11:48:21.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f3175686640 10 monclient(hunting): picked mon.noname-c con 0x7f3170142690 addr v1:172.21.15.98:6790/0 2024-03-29T11:48:21.251 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f3175686640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31700acad0 con 0x7f31701422b0 2024-03-29T11:48:21.251 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f3175686640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f317012da90 con 0x7f31700b17d0 2024-03-29T11:48:21.251 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f3175686640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31700ab5e0 con 0x7f3170142690 2024-03-29T11:48:21.251 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f3175686640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:21.251 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f3175686640 10 monclient(hunting): authenticate will time out at 808.318164s 2024-03-29T11:48:21.251 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f316e7fc640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f31701422b0 0x7f317012cc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:60934/0 (socket says 172.21.15.98:60934) 2024-03-29T11:48:21.251 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f316f7fe640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f3170142690 0x7f317012d380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:49658/0 (socket says 172.21.15.98:49658) 2024-03-29T11:48:21.251 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f316e7fc640 1 -- 172.21.15.98:0/114970647 learned_addr learned my addr 172.21.15.98:0/114970647 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:48:21.251 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 10 client.?.objecter ms_handle_connect 0x7f31701422b0 2024-03-29T11:48:21.251 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:21.251 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 10 client.?.objecter ms_handle_connect 0x7f3170142690 2024-03-29T11:48:21.251 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:21.251 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 10 client.?.objecter ms_handle_connect 0x7f31700b17d0 2024-03-29T11:48:21.251 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:21.251 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 1 -- 172.21.15.98:0/114970647 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1057976399 0 0) 0x7f31700acad0 con 0x7f31701422b0 2024-03-29T11:48:21.252 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 10 monclient(hunting): _negotiate 2024-03-29T11:48:21.252 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:21.252 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:21.252 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 10 monclient(hunting): my global_id is 4191 2024-03-29T11:48:21.252 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 1 -- 172.21.15.98:0/114970647 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f315c004eb0 con 0x7f31701422b0 2024-03-29T11:48:21.252 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 1 -- 172.21.15.98:0/114970647 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2847802469 0 0) 0x7f317012da90 con 0x7f31700b17d0 2024-03-29T11:48:21.252 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 10 monclient(hunting): _negotiate 2024-03-29T11:48:21.252 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:21.252 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:21.252 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 10 monclient(hunting): my global_id is 4198 2024-03-29T11:48:21.252 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 1 -- 172.21.15.98:0/114970647 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31700acad0 con 0x7f31700b17d0 2024-03-29T11:48:21.252 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 1 -- 172.21.15.98:0/114970647 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2578062044 0 0) 0x7f31700ab5e0 con 0x7f3170142690 2024-03-29T11:48:21.252 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 10 monclient(hunting): _negotiate 2024-03-29T11:48:21.252 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:21.252 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:21.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 10 monclient(hunting): my global_id is 4190 2024-03-29T11:48:21.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 1 -- 172.21.15.98:0/114970647 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f317012da90 con 0x7f3170142690 2024-03-29T11:48:21.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 1 -- 172.21.15.98:0/114970647 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4293709098 0 0) 0x7f315c004eb0 con 0x7f31701422b0 2024-03-29T11:48:21.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 1 -- 172.21.15.98:0/114970647 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31700ab5e0 con 0x7f31701422b0 2024-03-29T11:48:21.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 1 -- 172.21.15.98:0/114970647 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1531049984 0 0) 0x7f31700acad0 con 0x7f31700b17d0 2024-03-29T11:48:21.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 1 -- 172.21.15.98:0/114970647 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f315c004eb0 con 0x7f31700b17d0 2024-03-29T11:48:21.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 1 -- 172.21.15.98:0/114970647 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f3164001fa0 con 0x7f31701422b0 2024-03-29T11:48:21.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:21.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:48:21.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 10 monclient(hunting): dump: 2024-03-29T11:48:21.253 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:48:21.253 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:21.253 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:21.253 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:21.253 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:21.254 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:21.254 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:21.254 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:21.254 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:21.254 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:21.254 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:48:21.254 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 1 -- 172.21.15.98:0/114970647 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1999871617 0 0) 0x7f317012da90 con 0x7f3170142690 2024-03-29T11:48:21.254 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 1 -- 172.21.15.98:0/114970647 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31700acad0 con 0x7f3170142690 2024-03-29T11:48:21.254 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 1 -- 172.21.15.98:0/114970647 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f3168001150 con 0x7f31700b17d0 2024-03-29T11:48:21.254 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:21.254 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:48:21.254 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 10 monclient(hunting): dump: 2024-03-29T11:48:21.254 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:48:21.254 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:21.255 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:21.255 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:21.255 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:21.255 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:21.255 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:21.255 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:21.255 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:21.255 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:21.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 1 -- 172.21.15.98:0/114970647 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f3160001870 con 0x7f3170142690 2024-03-29T11:48:21.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:21.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:48:21.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 10 monclient(hunting): dump: 2024-03-29T11:48:21.255 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:48:21.255 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:21.255 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:21.256 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:21.256 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:21.256 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:21.256 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:21.256 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:21.256 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:21.256 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:21.256 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 1 -- 172.21.15.98:0/114970647 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 576835860 0 0) 0x7f31700ab5e0 con 0x7f31701422b0 2024-03-29T11:48:21.256 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 1 -- 172.21.15.98:0/114970647 >> v1:172.21.15.98:6790/0 conn(0x7f3170142690 legacy=0x7f317012d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:21.256 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 1 -- 172.21.15.98:0/114970647 >> v1:172.21.15.121:6789/0 conn(0x7f31700b17d0 legacy=0x7f317012e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:21.256 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 10 monclient: _finish_hunting 0 2024-03-29T11:48:21.256 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 1 monclient: found mon.a 2024-03-29T11:48:21.256 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:21.256 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:48:21.256 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 1 -- 172.21.15.98:0/114970647 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31700acd40 con 0x7f31701422b0 2024-03-29T11:48:21.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 10 monclient: _finish_auth 0 2024-03-29T11:48:21.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 10 monclient: _check_auth_tickets 2024-03-29T11:48:21.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f31577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:21.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f3175686640 5 monclient: authenticate success, global_id 4191 2024-03-29T11:48:21.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f3175686640 10 monclient: _renew_subs 2024-03-29T11:48:21.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f3175686640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:48:21.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f3175686640 1 -- 172.21.15.98:0/114970647 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31700ad020 con 0x7f31701422b0 2024-03-29T11:48:21.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f3175686640 10 client.4191.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:48:21.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f3175686640 10 monclient: _renew_subs 2024-03-29T11:48:21.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f3175686640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:48:21.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f3175686640 1 -- 172.21.15.98:0/114970647 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f31700b7000 con 0x7f31701422b0 2024-03-29T11:48:21.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.233+0000 7f3175686640 1 librados: init done 2024-03-29T11:48:21.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.234+0000 7f31577fe640 1 -- 172.21.15.98:0/114970647 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3164004e10 con 0x7f31701422b0 2024-03-29T11:48:21.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.234+0000 7f31577fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:21.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.234+0000 7f31577fe640 1 -- 172.21.15.98:0/114970647 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f3164002100 con 0x7f31701422b0 2024-03-29T11:48:21.258 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.234+0000 7f31577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:21.258 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.234+0000 7f31577fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:48:21.258 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.234+0000 7f316d7fa640 4 set_mon_vals no callback set 2024-03-29T11:48:21.258 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.234+0000 7f31577fe640 10 monclient: dump: 2024-03-29T11:48:21.258 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:48:21.258 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:21.258 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:21.258 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:21.258 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:21.258 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:21.258 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:21.258 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:21.258 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:21.258 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:21.258 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.234+0000 7f31577fe640 1 -- 172.21.15.98:0/114970647 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f31640055f0 con 0x7f31701422b0 2024-03-29T11:48:21.259 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.234+0000 7f31577fe640 1 -- 172.21.15.98:0/114970647 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f3164006820 con 0x7f31701422b0 2024-03-29T11:48:21.259 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.234+0000 7f31577fe640 10 client.4191.objecter ms_dispatch 0x7f3170000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:48:21.259 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.234+0000 7f31577fe640 3 client.4191.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:48:21.259 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.234+0000 7f31577fe640 3 client.4191.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:48:21.259 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.234+0000 7f31577fe640 20 client.4191.objecter dump_active .. 0 homeless 2024-03-29T11:48:21.259 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.235+0000 7f3175686640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:21.259 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.235+0000 7f3175686640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:21.259 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.235+0000 7f3175686640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:48:21.259 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.235+0000 7f3175686640 1 -- 172.21.15.98:0/114970647 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3150005080 con 0x7f31701422b0 2024-03-29T11:48:21.259 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.237+0000 7f31577fe640 1 -- 172.21.15.98:0/114970647 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f3164006ab0 con 0x7f31701422b0 2024-03-29T11:48:21.259 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.237+0000 7f31577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:21.259 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.237+0000 7f31577fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:48:21.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.299+0000 7f3175686640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:21.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.299+0000 7f3175686640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:21.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.299+0000 7f3175686640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:48:21.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.299+0000 7f3175686640 1 -- 172.21.15.98:0/114970647 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3150005330 con 0x7f31701422b0 2024-03-29T11:48:21.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.299+0000 7f31577fe640 1 -- 172.21.15.98:0/114970647 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f31640058a0 con 0x7f31701422b0 2024-03-29T11:48:21.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.299+0000 7f31577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:21.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.299+0000 7f31577fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:48:21.305 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:48:21.305 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:48:21.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.300+0000 7f3175686640 10 librados: watch_flush enter 2024-03-29T11:48:21.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.300+0000 7f3175686640 10 librados: watch_flush exit 2024-03-29T11:48:21.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.300+0000 7f3175686640 20 client.4191.objecter shutdown clearing up homeless session... 2024-03-29T11:48:21.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.300+0000 7f3175686640 10 client.4191.objecter successfully canceled tick 2024-03-29T11:48:21.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.300+0000 7f3175686640 10 monclient: shutdown 2024-03-29T11:48:21.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.300+0000 7f3175686640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:21.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.300+0000 7f3175686640 1 -- 172.21.15.98:0/114970647 >> v1:172.21.15.98:6789/0 conn(0x7f31701422b0 legacy=0x7f317012cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:21.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.300+0000 7f3175686640 1 -- 172.21.15.98:0/114970647 shutdown_connections 2024-03-29T11:48:21.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.300+0000 7f3175686640 1 -- 172.21.15.98:0/114970647 >> 172.21.15.98:0/114970647 conn(0x7f3170066a00 msgr2=0x7f317013d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:21.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.300+0000 7f3175686640 1 -- 172.21.15.98:0/114970647 shutdown_connections 2024-03-29T11:48:21.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.300+0000 7f3175686640 1 -- 172.21.15.98:0/114970647 wait complete. 2024-03-29T11:48:21.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:21.300+0000 7f3175686640 1 librados: shutdown 2024-03-29T11:48:21.315 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:48:21.563 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T11:48:21.557+0000 7faaea042040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:48:21.657 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:48:21.651+0000 7f4e33c10040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:48:21.838 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T11:48:21.833+0000 7faaea042040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:48:21.890 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T11:48:21.885+0000 7faaea042040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:48:21.932 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:48:21.927+0000 7f4e33c10040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:48:21.983 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:48:21.977+0000 7f4e33c10040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:48:22.004 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T11:48:21.999+0000 7faaea042040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:48:22.103 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:48:22.097+0000 7f4e33c10040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:48:23.021 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T11:48:23.015+0000 7faa8ad51640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T11:48:23.021 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T11:48:23.015+0000 7faa8ad51640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T11:48:24.198 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-03-29T11:48:24.192+0000 7fb0655bd640 -1 osd.3 0 log_to_monitors true 2024-03-29T11:48:24.212 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-03-29T11:48:24.206+0000 7fc352bfa640 -1 osd.4 0 log_to_monitors true 2024-03-29T11:48:24.238 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-03-29T11:48:24.232+0000 7f57dcb9a640 -1 osd.7 0 log_to_monitors true 2024-03-29T11:48:24.238 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-03-29T11:48:24.233+0000 7ff874e11640 -1 osd.6 0 log_to_monitors true 2024-03-29T11:48:24.239 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-03-29T11:48:24.233+0000 7f4f9216b640 -1 osd.0 0 log_to_monitors true 2024-03-29T11:48:24.240 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-03-29T11:48:24.234+0000 7f446d217640 -1 osd.1 0 log_to_monitors true 2024-03-29T11:48:24.263 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-03-29T11:48:24.257+0000 7f1ed1a37640 -1 osd.5 0 log_to_monitors true 2024-03-29T11:48:24.268 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-03-29T11:48:24.263+0000 7f96c4611640 -1 osd.2 0 log_to_monitors true 2024-03-29T11:48:28.122 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:48:28.269 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.263+0000 7fc286347640 10 monclient: get_monmap_and_config 2024-03-29T11:48:28.269 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.263+0000 7fc286347640 10 monclient: build_initial_monmap 2024-03-29T11:48:28.269 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.263+0000 7fc286347640 10 monclient: monmap: 2024-03-29T11:48:28.269 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:48:28.269 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:28.270 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:48:28.264580+0000 2024-03-29T11:48:28.270 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:48:28.264580+0000 2024-03-29T11:48:28.270 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:28.270 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:28.270 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:48:28.270 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:48:28.270 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:48:28.270 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:28.270 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.264+0000 7fc286347640 1 Processor -- start 2024-03-29T11:48:28.271 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.265+0000 7fc286347640 1 -- start start 2024-03-29T11:48:28.271 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.265+0000 7fc286347640 10 monclient: init 2024-03-29T11:48:28.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.266+0000 7fc286347640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:28.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.266+0000 7fc286347640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:48:28.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.266+0000 7fc286347640 10 monclient(hunting): picked mon.noname-b con 0x7fc2801457f0 addr v1:172.21.15.98:6790/0 2024-03-29T11:48:28.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.266+0000 7fc286347640 10 monclient(hunting): picked mon.noname-c con 0x7fc28013f650 addr v1:172.21.15.121:6789/0 2024-03-29T11:48:28.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.266+0000 7fc286347640 10 monclient(hunting): picked mon.noname-a con 0x7fc2800a9d70 addr v1:172.21.15.98:6789/0 2024-03-29T11:48:28.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.266+0000 7fc27f7fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fc28013f650 0x7fc2800b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:36928/0 (socket says 172.21.15.98:36928) 2024-03-29T11:48:28.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.266+0000 7fc27f7fe640 1 -- 172.21.15.98:0/3856611901 learned_addr learned my addr 172.21.15.98:0/3856611901 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:48:28.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.266+0000 7fc286347640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2800aaef0 con 0x7fc2800a9d70 2024-03-29T11:48:28.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.266+0000 7fc286347640 1 -- 172.21.15.98:0/3856611901 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc280145bd0 con 0x7fc28013f650 2024-03-29T11:48:28.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.266+0000 7fc286347640 1 -- 172.21.15.98:0/3856611901 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc280145eb0 con 0x7fc2801457f0 2024-03-29T11:48:28.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.266+0000 7fc286347640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:28.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.266+0000 7fc286347640 10 monclient(hunting): authenticate will time out at 815.351557s 2024-03-29T11:48:28.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.266+0000 7fc27effd640 1 -- 172.21.15.98:0/3856611901 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2238538389 0 0) 0x7fc280145bd0 con 0x7fc28013f650 2024-03-29T11:48:28.274 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.266+0000 7fc27effd640 10 monclient(hunting): _negotiate 2024-03-29T11:48:28.274 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.266+0000 7fc27effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:28.274 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.266+0000 7fc27effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:28.274 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.266+0000 7fc27effd640 10 monclient(hunting): my global_id is 4243 2024-03-29T11:48:28.274 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.266+0000 7fc27effd640 1 -- 172.21.15.98:0/3856611901 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc26c003520 con 0x7fc28013f650 2024-03-29T11:48:28.274 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.267+0000 7fc27effd640 1 -- 172.21.15.98:0/3856611901 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2464275206 0 0) 0x7fc26c003520 con 0x7fc28013f650 2024-03-29T11:48:28.274 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.267+0000 7fc27effd640 1 -- 172.21.15.98:0/3856611901 >> v1:172.21.15.98:6790/0 conn(0x7fc2801457f0 legacy=0x7fc28013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:28.274 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.267+0000 7fc27effd640 1 -- 172.21.15.98:0/3856611901 >> v1:172.21.15.98:6789/0 conn(0x7fc2800a9d70 legacy=0x7fc28014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:48:28.274 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.267+0000 7fc27effd640 10 monclient: _finish_hunting 0 2024-03-29T11:48:28.274 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.267+0000 7fc27effd640 1 monclient: found mon.noname-c 2024-03-29T11:48:28.274 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.267+0000 7fc27effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:28.274 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.267+0000 7fc27effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:48:28.274 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.267+0000 7fc27effd640 1 -- 172.21.15.98:0/3856611901 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc280146340 con 0x7fc28013f650 2024-03-29T11:48:28.274 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.267+0000 7fc27effd640 10 monclient: _finish_auth 0 2024-03-29T11:48:28.274 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.267+0000 7fc27effd640 10 monclient: _check_auth_tickets 2024-03-29T11:48:28.275 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.267+0000 7fc27effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:28.275 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.267+0000 7fc27effd640 1 -- 172.21.15.98:0/3856611901 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fc270001690 con 0x7fc28013f650 2024-03-29T11:48:28.275 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.267+0000 7fc27effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:28.275 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.267+0000 7fc27effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:48:28.275 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.267+0000 7fc27effd640 10 monclient: dump: 2024-03-29T11:48:28.275 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:48:28.275 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:28.275 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:28.275 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:28.275 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:28.275 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:28.275 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:28.275 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:28.275 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:28.275 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:28.276 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.267+0000 7fc286347640 5 monclient: authenticate success, global_id 4243 2024-03-29T11:48:28.276 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.267+0000 7fc286347640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:48:28.281 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.268+0000 7fc27effd640 1 -- 172.21.15.98:0/3856611901 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc270001860 con 0x7fc28013f650 2024-03-29T11:48:28.281 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.268+0000 7fc27effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:28.282 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.268+0000 7fc286347640 10 monclient: get_monmap_and_config success 2024-03-29T11:48:28.282 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.268+0000 7fc27effd640 1 -- 172.21.15.98:0/3856611901 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fc270001cb0 con 0x7fc28013f650 2024-03-29T11:48:28.282 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.268+0000 7fc27effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:28.282 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.268+0000 7fc286347640 4 set_mon_vals no callback set 2024-03-29T11:48:28.282 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.268+0000 7fc27effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:48:28.282 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.268+0000 7fc27effd640 10 monclient: dump: 2024-03-29T11:48:28.282 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:48:28.282 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:28.282 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:28.282 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:28.282 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:28.282 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:28.282 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:28.282 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:28.283 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:28.283 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:28.283 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.268+0000 7fc286347640 10 monclient: shutdown 2024-03-29T11:48:28.283 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.268+0000 7fc286347640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:28.283 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.268+0000 7fc286347640 1 -- 172.21.15.98:0/3856611901 >> v1:172.21.15.121:6789/0 conn(0x7fc28013f650 legacy=0x7fc2800b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:28.283 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.269+0000 7fc286347640 1 -- 172.21.15.98:0/3856611901 shutdown_connections 2024-03-29T11:48:28.283 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.269+0000 7fc286347640 1 -- 172.21.15.98:0/3856611901 >> 172.21.15.98:0/3856611901 conn(0x7fc280066a00 msgr2=0x7fc280066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:28.283 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.269+0000 7fc286347640 1 -- 172.21.15.98:0/3856611901 shutdown_connections 2024-03-29T11:48:28.283 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.269+0000 7fc286347640 1 -- 172.21.15.98:0/3856611901 wait complete. 2024-03-29T11:48:28.283 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.270+0000 7fc286347640 10 monclient: build_initial_monmap 2024-03-29T11:48:28.283 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.270+0000 7fc286347640 10 monclient: monmap: 2024-03-29T11:48:28.283 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:48:28.283 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:48:28.283 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:48:28.283 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:48:28.284 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:28.284 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:28.284 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:48:28.284 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:48:28.284 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:48:28.284 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:28.284 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.271+0000 7fc286347640 1 librados: starting msgr at 2024-03-29T11:48:28.284 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.271+0000 7fc286347640 1 librados: starting objecter 2024-03-29T11:48:28.284 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.271+0000 7fc286347640 1 Processor -- start 2024-03-29T11:48:28.284 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.271+0000 7fc286347640 1 -- start start 2024-03-29T11:48:28.284 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.271+0000 7fc286347640 1 librados: setting wanted keys 2024-03-29T11:48:28.284 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.271+0000 7fc286347640 1 librados: calling monclient init 2024-03-29T11:48:28.284 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.271+0000 7fc286347640 10 monclient: init 2024-03-29T11:48:28.284 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.272+0000 7fc286347640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:28.284 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.272+0000 7fc286347640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:48:28.285 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.272+0000 7fc286347640 10 monclient(hunting): picked mon.noname-a con 0x7fc2800a9d70 addr v1:172.21.15.98:6789/0 2024-03-29T11:48:28.285 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.272+0000 7fc286347640 10 monclient(hunting): picked mon.noname-c con 0x7fc28013f650 addr v1:172.21.15.98:6790/0 2024-03-29T11:48:28.285 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.272+0000 7fc286347640 10 monclient(hunting): picked mon.noname-b con 0x7fc2801457f0 addr v1:172.21.15.121:6789/0 2024-03-29T11:48:28.285 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.272+0000 7fc286347640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2801daba0 con 0x7fc2800a9d70 2024-03-29T11:48:28.285 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.272+0000 7fc286347640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2801d4ae0 con 0x7fc2801457f0 2024-03-29T11:48:28.285 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.272+0000 7fc286347640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc280141800 con 0x7fc28013f650 2024-03-29T11:48:28.285 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.272+0000 7fc286347640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:28.285 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.272+0000 7fc286347640 10 monclient(hunting): authenticate will time out at 815.357812s 2024-03-29T11:48:28.285 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.273+0000 7fc27ffff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc2800a9d70 0x7fc2801da490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:58148/0 (socket says 172.21.15.98:58148) 2024-03-29T11:48:28.285 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.273+0000 7fc27ffff640 1 -- 172.21.15.98:0/4180333405 learned_addr learned my addr 172.21.15.98:0/4180333405 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:48:28.285 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.273+0000 7fc27cff9640 10 client.?.objecter ms_handle_connect 0x7fc2801457f0 2024-03-29T11:48:28.285 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.273+0000 7fc27cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:28.285 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.273+0000 7fc27cff9640 10 client.?.objecter ms_handle_connect 0x7fc2800a9d70 2024-03-29T11:48:28.285 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.273+0000 7fc27cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:28.285 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.273+0000 7fc27cff9640 10 client.?.objecter ms_handle_connect 0x7fc28013f650 2024-03-29T11:48:28.286 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.273+0000 7fc27cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:28.286 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.273+0000 7fc27ffff640 1 -- 172.21.15.98:0/4180333405 >> v1:172.21.15.98:6789/0 conn(0x7fc2800a9d70 legacy=0x7fc2801da490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-29T11:48:28.286 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.273+0000 7fc27ffff640 1 -- 172.21.15.98:0/4180333405 >> v1:172.21.15.98:6789/0 conn(0x7fc2800a9d70 legacy=0x7fc2801da490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:48:28.286 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.273+0000 7fc27ffff640 1 --1- 172.21.15.98:0/4180333405 >> v1:172.21.15.98:6789/0 conn(0x7fc2800a9d70 0x7fc2801da490 :-1 s=OPENED pgs=20916 cs=1 l=1).handle_message read tag failed 2024-03-29T11:48:28.286 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.273+0000 7fc27ffff640 1 --1- 172.21.15.98:0/4180333405 >> v1:172.21.15.98:6789/0 conn(0x7fc2800a9d70 0x7fc2801da490 :-1 s=OPENED pgs=20916 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:48:28.286 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.273+0000 7fc27cff9640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.98:6789/0 2024-03-29T11:48:28.286 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.274+0000 7fc27cff9640 1 -- 172.21.15.98:0/4180333405 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 419844983 0 0) 0x7fc2801d4ae0 con 0x7fc2801457f0 2024-03-29T11:48:28.286 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.274+0000 7fc27cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:48:28.286 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.274+0000 7fc27cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:28.286 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.274+0000 7fc27cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:28.286 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.274+0000 7fc27cff9640 10 monclient(hunting): my global_id is 4246 2024-03-29T11:48:28.286 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.274+0000 7fc27cff9640 1 -- 172.21.15.98:0/4180333405 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc260004eb0 con 0x7fc2801457f0 2024-03-29T11:48:28.286 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.274+0000 7fc27cff9640 1 -- 172.21.15.98:0/4180333405 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 663265641 0 0) 0x7fc280141800 con 0x7fc28013f650 2024-03-29T11:48:28.286 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.274+0000 7fc27cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:48:28.287 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.274+0000 7fc27cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:28.287 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.274+0000 7fc27cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:28.287 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.274+0000 7fc27cff9640 10 monclient(hunting): my global_id is 4232 2024-03-29T11:48:28.287 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.274+0000 7fc27cff9640 1 -- 172.21.15.98:0/4180333405 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2801d4ae0 con 0x7fc28013f650 2024-03-29T11:48:28.287 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.274+0000 7fc27cff9640 1 -- 172.21.15.98:0/4180333405 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 577961634 0 0) 0x7fc2801d4ae0 con 0x7fc28013f650 2024-03-29T11:48:28.287 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.274+0000 7fc27cff9640 1 -- 172.21.15.98:0/4180333405 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc280141800 con 0x7fc28013f650 2024-03-29T11:48:28.287 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.274+0000 7fc27cff9640 1 -- 172.21.15.98:0/4180333405 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fc2700019d0 con 0x7fc28013f650 2024-03-29T11:48:28.287 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.274+0000 7fc27cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:28.287 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.274+0000 7fc27cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:48:28.287 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.274+0000 7fc27cff9640 10 monclient(hunting): dump: 2024-03-29T11:48:28.287 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:48:28.287 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:28.287 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:28.287 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:28.288 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:28.288 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:28.288 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:28.288 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:28.288 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:28.288 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:28.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.274+0000 7fc27cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:48:28.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.275+0000 7fc27cff9640 1 -- 172.21.15.98:0/4180333405 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1567801353 0 0) 0x7fc260004eb0 con 0x7fc2801457f0 2024-03-29T11:48:28.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.275+0000 7fc27cff9640 1 -- 172.21.15.98:0/4180333405 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2801d4ae0 con 0x7fc2801457f0 2024-03-29T11:48:28.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.275+0000 7fc27cff9640 1 -- 172.21.15.98:0/4180333405 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fc278001890 con 0x7fc2801457f0 2024-03-29T11:48:28.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.275+0000 7fc27cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:28.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.275+0000 7fc27cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:48:28.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.275+0000 7fc27cff9640 10 monclient(hunting): dump: 2024-03-29T11:48:28.288 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:48:28.288 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:28.289 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:28.289 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:28.289 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:28.289 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:28.289 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:28.289 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:28.289 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:28.289 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:28.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.275+0000 7fc27cff9640 1 -- 172.21.15.98:0/4180333405 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3615003102 0 0) 0x7fc280141800 con 0x7fc28013f650 2024-03-29T11:48:28.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.275+0000 7fc27cff9640 1 -- 172.21.15.98:0/4180333405 >> v1:172.21.15.121:6789/0 conn(0x7fc2801457f0 legacy=0x7fc28013afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:28.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.275+0000 7fc27cff9640 1 -- 172.21.15.98:0/4180333405 >> v1:172.21.15.98:6789/0 conn(0x7fc2800a9d70 legacy=0x7fc2801da490 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:48:28.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.275+0000 7fc27cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:48:28.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.275+0000 7fc27cff9640 1 monclient: found mon.c 2024-03-29T11:48:28.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.275+0000 7fc27cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:28.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.275+0000 7fc27cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:48:28.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.275+0000 7fc27cff9640 1 -- 172.21.15.98:0/4180333405 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc280141990 con 0x7fc28013f650 2024-03-29T11:48:28.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.275+0000 7fc27cff9640 10 monclient: _finish_auth 0 2024-03-29T11:48:28.292 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.275+0000 7fc27cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:48:28.293 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.275+0000 7fc27cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:28.293 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.275+0000 7fc286347640 5 monclient: authenticate success, global_id 4232 2024-03-29T11:48:28.293 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.275+0000 7fc286347640 10 monclient: _renew_subs 2024-03-29T11:48:28.293 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.275+0000 7fc286347640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:48:28.293 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.275+0000 7fc286347640 1 -- 172.21.15.98:0/4180333405 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc280138a90 con 0x7fc28013f650 2024-03-29T11:48:28.293 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.275+0000 7fc286347640 10 client.4232.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:48:28.293 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.275+0000 7fc286347640 10 monclient: _renew_subs 2024-03-29T11:48:28.293 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.275+0000 7fc286347640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:48:28.293 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.275+0000 7fc286347640 1 -- 172.21.15.98:0/4180333405 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc280139b60 con 0x7fc28013f650 2024-03-29T11:48:28.293 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.275+0000 7fc286347640 1 librados: init done 2024-03-29T11:48:28.293 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.276+0000 7fc27cff9640 1 -- 172.21.15.98:0/4180333405 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc270004940 con 0x7fc28013f650 2024-03-29T11:48:28.293 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.276+0000 7fc27cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:28.293 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.277+0000 7fc27e7fc640 4 set_mon_vals no callback set 2024-03-29T11:48:28.294 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.276+0000 7fc27cff9640 1 -- 172.21.15.98:0/4180333405 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fc270004c50 con 0x7fc28013f650 2024-03-29T11:48:28.294 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.277+0000 7fc27cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:28.294 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.277+0000 7fc27cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:48:28.294 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.277+0000 7fc27cff9640 10 monclient: dump: 2024-03-29T11:48:28.294 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:48:28.294 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:28.294 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:28.294 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:28.294 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:28.294 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:28.294 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:28.294 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:28.294 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:28.294 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:28.294 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.279+0000 7fc286347640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:28.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.279+0000 7fc286347640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:28.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.279+0000 7fc286347640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:48:28.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.279+0000 7fc27cff9640 1 -- 172.21.15.98:0/4180333405 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7fc27001ac20 con 0x7fc28013f650 2024-03-29T11:48:28.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.279+0000 7fc286347640 1 -- 172.21.15.98:0/4180333405 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc264005080 con 0x7fc28013f650 2024-03-29T11:48:28.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.280+0000 7fc27cff9640 1 -- 172.21.15.98:0/4180333405 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7fc270084a10 con 0x7fc28013f650 2024-03-29T11:48:28.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.280+0000 7fc27cff9640 10 client.4232.objecter ms_dispatch 0x7fc280000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:48:28.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.280+0000 7fc27cff9640 3 client.4232.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:48:28.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.280+0000 7fc27cff9640 3 client.4232.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:48:28.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.280+0000 7fc27cff9640 20 client.4232.objecter dump_active .. 0 homeless 2024-03-29T11:48:28.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.280+0000 7fc27ffff640 10 monclient: get_auth_request con 0x7fc26006de80 auth_method 0 2024-03-29T11:48:28.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.281+0000 7fc27cff9640 10 client.4232.objecter ms_handle_connect 0x7fc26006de80 2024-03-29T11:48:28.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.292+0000 7fc27cff9640 1 -- 172.21.15.98:0/4180333405 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc2700502f0 con 0x7fc28013f650 2024-03-29T11:48:28.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.292+0000 7fc27cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:28.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.292+0000 7fc27cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:48:28.672 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.667+0000 7fc286347640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:28.673 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.667+0000 7fc286347640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:28.673 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.667+0000 7fc286347640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:48:28.673 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.667+0000 7fc286347640 1 -- 172.21.15.98:0/4180333405 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc264005330 con 0x7fc28013f650 2024-03-29T11:48:28.673 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.668+0000 7fc27cff9640 1 -- 172.21.15.98:0/4180333405 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7fc270052220 con 0x7fc28013f650 2024-03-29T11:48:28.673 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.668+0000 7fc27cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:28.673 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.668+0000 7fc27cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:48:28.674 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:48:28.674 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:48:28.678 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.672+0000 7fc286347640 10 librados: watch_flush enter 2024-03-29T11:48:28.678 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.672+0000 7fc286347640 10 librados: watch_flush exit 2024-03-29T11:48:28.678 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.672+0000 7fc286347640 20 client.4232.objecter shutdown clearing up homeless session... 2024-03-29T11:48:28.678 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.672+0000 7fc286347640 10 client.4232.objecter successfully canceled tick 2024-03-29T11:48:28.678 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.672+0000 7fc286347640 1 -- 172.21.15.98:0/4180333405 >> v1:172.21.15.98:6816/4065186974 conn(0x7fc26006de80 legacy=0x7fc260070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:28.678 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.672+0000 7fc286347640 10 monclient: shutdown 2024-03-29T11:48:28.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.672+0000 7fc286347640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:28.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.672+0000 7fc286347640 1 -- 172.21.15.98:0/4180333405 >> v1:172.21.15.98:6790/0 conn(0x7fc28013f650 legacy=0x7fc2801d43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:28.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.673+0000 7fc286347640 1 -- 172.21.15.98:0/4180333405 shutdown_connections 2024-03-29T11:48:28.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.673+0000 7fc286347640 1 -- 172.21.15.98:0/4180333405 >> 172.21.15.98:0/4180333405 conn(0x7fc280066a00 msgr2=0x7fc2800b4790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:28.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.673+0000 7fc286347640 1 -- 172.21.15.98:0/4180333405 shutdown_connections 2024-03-29T11:48:28.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.673+0000 7fc286347640 1 -- 172.21.15.98:0/4180333405 wait complete. 2024-03-29T11:48:28.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:28.673+0000 7fc286347640 1 librados: shutdown 2024-03-29T11:48:28.694 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:48:35.500 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:48:35.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.627+0000 7fd0c2b97640 10 monclient: get_monmap_and_config 2024-03-29T11:48:35.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.627+0000 7fd0c2b97640 10 monclient: build_initial_monmap 2024-03-29T11:48:35.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.627+0000 7fd0c2b97640 10 monclient: monmap: 2024-03-29T11:48:35.633 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:48:35.633 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:35.633 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:48:35.628060+0000 2024-03-29T11:48:35.633 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:48:35.628060+0000 2024-03-29T11:48:35.633 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:35.633 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:35.633 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:48:35.633 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:48:35.633 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:48:35.633 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:35.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.628+0000 7fd0c2b97640 1 Processor -- start 2024-03-29T11:48:35.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.628+0000 7fd0c2b97640 1 -- start start 2024-03-29T11:48:35.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.628+0000 7fd0c2b97640 10 monclient: init 2024-03-29T11:48:35.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.628+0000 7fd0c2b97640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:35.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.629+0000 7fd0c2b97640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:48:35.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.629+0000 7fd0c2b97640 10 monclient(hunting): picked mon.noname-b con 0x7fd0bc1422b0 addr v1:172.21.15.98:6790/0 2024-03-29T11:48:35.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.629+0000 7fd0c2b97640 10 monclient(hunting): picked mon.noname-c con 0x7fd0bc142690 addr v1:172.21.15.121:6789/0 2024-03-29T11:48:35.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.629+0000 7fd0c2b97640 10 monclient(hunting): picked mon.noname-a con 0x7fd0bc0b17d0 addr v1:172.21.15.98:6789/0 2024-03-29T11:48:35.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.629+0000 7fd0c2b97640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0bc142c80 con 0x7fd0bc0b17d0 2024-03-29T11:48:35.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.629+0000 7fd0c2b97640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0bc0aae40 con 0x7fd0bc142690 2024-03-29T11:48:35.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.629+0000 7fd0c2b97640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0bc13fbd0 con 0x7fd0bc1422b0 2024-03-29T11:48:35.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.629+0000 7fd0c2b97640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:35.636 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.629+0000 7fd0c2b97640 10 monclient(hunting): authenticate will time out at 822.714352s 2024-03-29T11:48:35.636 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.629+0000 7fd0c0933640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fd0bc1422b0 0x7fd0bc13f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:43372/0 (socket says 172.21.15.98:43372) 2024-03-29T11:48:35.636 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.629+0000 7fd0c0933640 1 -- 172.21.15.98:0/130558436 learned_addr learned my addr 172.21.15.98:0/130558436 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:48:35.636 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.629+0000 7fd0bb7fe640 1 -- 172.21.15.98:0/130558436 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1482560746 0 0) 0x7fd0bc13fbd0 con 0x7fd0bc1422b0 2024-03-29T11:48:35.636 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.629+0000 7fd0bb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:48:35.636 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.629+0000 7fd0bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:35.636 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.629+0000 7fd0bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:35.636 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.629+0000 7fd0bb7fe640 10 monclient(hunting): my global_id is 4235 2024-03-29T11:48:35.636 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.629+0000 7fd0bb7fe640 1 -- 172.21.15.98:0/130558436 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0ac003520 con 0x7fd0bc1422b0 2024-03-29T11:48:35.637 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.630+0000 7fd0bb7fe640 1 -- 172.21.15.98:0/130558436 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 172496251 0 0) 0x7fd0bc0aae40 con 0x7fd0bc142690 2024-03-29T11:48:35.637 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.630+0000 7fd0bb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:48:35.637 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.630+0000 7fd0bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:35.637 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.630+0000 7fd0bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:35.638 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.630+0000 7fd0bb7fe640 10 monclient(hunting): my global_id is 4249 2024-03-29T11:48:35.638 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.630+0000 7fd0bb7fe640 1 -- 172.21.15.98:0/130558436 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0bc13fbd0 con 0x7fd0bc142690 2024-03-29T11:48:35.638 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.630+0000 7fd0bb7fe640 1 -- 172.21.15.98:0/130558436 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3487050945 0 0) 0x7fd0ac003520 con 0x7fd0bc1422b0 2024-03-29T11:48:35.638 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.630+0000 7fd0bb7fe640 1 -- 172.21.15.98:0/130558436 >> v1:172.21.15.121:6789/0 conn(0x7fd0bc142690 legacy=0x7fd0bc0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:35.638 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.630+0000 7fd0bb7fe640 1 -- 172.21.15.98:0/130558436 >> v1:172.21.15.98:6789/0 conn(0x7fd0bc0b17d0 legacy=0x7fd0bc0b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:48:35.638 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.631+0000 7fd0bb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:48:35.638 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.631+0000 7fd0bb7fe640 1 monclient: found mon.noname-b 2024-03-29T11:48:35.638 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.631+0000 7fd0bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:35.638 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.631+0000 7fd0bb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:48:35.638 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.631+0000 7fd0bb7fe640 1 -- 172.21.15.98:0/130558436 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0bc151940 con 0x7fd0bc1422b0 2024-03-29T11:48:35.638 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.631+0000 7fd0bb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:48:35.638 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.631+0000 7fd0bb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:48:35.638 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.631+0000 7fd0bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:35.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.631+0000 7fd0bb7fe640 1 -- 172.21.15.98:0/130558436 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fd0b0001690 con 0x7fd0bc1422b0 2024-03-29T11:48:35.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.631+0000 7fd0c2b97640 5 monclient: authenticate success, global_id 4235 2024-03-29T11:48:35.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.631+0000 7fd0c2b97640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:48:35.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.631+0000 7fd0bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:35.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.631+0000 7fd0bb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:48:35.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.631+0000 7fd0bb7fe640 10 monclient: dump: 2024-03-29T11:48:35.639 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:48:35.639 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:35.639 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:35.639 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:35.639 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:35.639 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:35.639 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:35.639 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:35.639 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:35.640 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:35.640 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.631+0000 7fd0c2b97640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:48:35.640 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.632+0000 7fd0bb7fe640 1 -- 172.21.15.98:0/130558436 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd0b00017f0 con 0x7fd0bc1422b0 2024-03-29T11:48:35.640 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.632+0000 7fd0bb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:35.640 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.632+0000 7fd0bb7fe640 1 -- 172.21.15.98:0/130558436 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fd0b0001b80 con 0x7fd0bc1422b0 2024-03-29T11:48:35.640 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.632+0000 7fd0bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:35.640 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.632+0000 7fd0bb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:48:35.640 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.632+0000 7fd0bb7fe640 10 monclient: dump: 2024-03-29T11:48:35.640 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:48:35.640 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:35.640 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:35.640 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:35.640 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:35.640 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:35.640 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:35.640 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:35.641 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:35.641 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:35.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.632+0000 7fd0c2b97640 10 monclient: get_monmap_and_config success 2024-03-29T11:48:35.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.632+0000 7fd0c2b97640 4 set_mon_vals no callback set 2024-03-29T11:48:35.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.632+0000 7fd0c2b97640 10 monclient: shutdown 2024-03-29T11:48:35.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.632+0000 7fd0c2b97640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:35.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.632+0000 7fd0c2b97640 1 -- 172.21.15.98:0/130558436 >> v1:172.21.15.98:6790/0 conn(0x7fd0bc1422b0 legacy=0x7fd0bc13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:35.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.633+0000 7fd0c2b97640 1 -- 172.21.15.98:0/130558436 shutdown_connections 2024-03-29T11:48:35.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.633+0000 7fd0c2b97640 1 -- 172.21.15.98:0/130558436 >> 172.21.15.98:0/130558436 conn(0x7fd0bc066a00 msgr2=0x7fd0bc066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:35.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.633+0000 7fd0c2b97640 1 -- 172.21.15.98:0/130558436 shutdown_connections 2024-03-29T11:48:35.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.633+0000 7fd0c2b97640 1 -- 172.21.15.98:0/130558436 wait complete. 2024-03-29T11:48:35.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.634+0000 7fd0c2b97640 10 monclient: build_initial_monmap 2024-03-29T11:48:35.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.634+0000 7fd0c2b97640 10 monclient: monmap: 2024-03-29T11:48:35.649 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:48:35.649 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:48:35.649 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:48:35.649 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:48:35.649 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:35.649 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:35.649 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:48:35.649 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:48:35.649 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:48:35.650 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:35.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.635+0000 7fd0c2b97640 1 librados: starting msgr at 2024-03-29T11:48:35.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.635+0000 7fd0c2b97640 1 librados: starting objecter 2024-03-29T11:48:35.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.635+0000 7fd0c2b97640 1 Processor -- start 2024-03-29T11:48:35.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.635+0000 7fd0c2b97640 1 -- start start 2024-03-29T11:48:35.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.635+0000 7fd0c2b97640 1 librados: setting wanted keys 2024-03-29T11:48:35.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.635+0000 7fd0c2b97640 1 librados: calling monclient init 2024-03-29T11:48:35.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.635+0000 7fd0c2b97640 10 monclient: init 2024-03-29T11:48:35.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.636+0000 7fd0c2b97640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:35.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.636+0000 7fd0c2b97640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:48:35.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.637+0000 7fd0c2b97640 10 monclient(hunting): picked mon.noname-b con 0x7fd0bc0b17d0 addr v1:172.21.15.121:6789/0 2024-03-29T11:48:35.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.637+0000 7fd0c2b97640 10 monclient(hunting): picked mon.noname-a con 0x7fd0bc1422b0 addr v1:172.21.15.98:6789/0 2024-03-29T11:48:35.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.637+0000 7fd0c2b97640 10 monclient(hunting): picked mon.noname-c con 0x7fd0bc142690 addr v1:172.21.15.98:6790/0 2024-03-29T11:48:35.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.637+0000 7fd0c2b97640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0bc0ad130 con 0x7fd0bc1422b0 2024-03-29T11:48:35.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.637+0000 7fd0c2b97640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0bc1324d0 con 0x7fd0bc0b17d0 2024-03-29T11:48:35.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.637+0000 7fd0c2b97640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0bc1320c0 con 0x7fd0bc142690 2024-03-29T11:48:35.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.637+0000 7fd0c2b97640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:35.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.637+0000 7fd0c2b97640 10 monclient(hunting): authenticate will time out at 822.722862s 2024-03-29T11:48:35.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.637+0000 7fd0c0933640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fd0bc0b17d0 0x7fd0bc130080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:42110/0 (socket says 172.21.15.98:42110) 2024-03-29T11:48:35.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.637+0000 7fd0c0933640 1 -- 172.21.15.98:0/2740598096 learned_addr learned my addr 172.21.15.98:0/2740598096 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:48:35.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.638+0000 7fd0b97fa640 10 client.?.objecter ms_handle_connect 0x7fd0bc0b17d0 2024-03-29T11:48:35.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.638+0000 7fd0b97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:35.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.638+0000 7fd0b97fa640 10 client.?.objecter ms_handle_connect 0x7fd0bc142690 2024-03-29T11:48:35.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.638+0000 7fd0b97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:35.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.638+0000 7fd0b97fa640 1 -- 172.21.15.98:0/2740598096 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1571573906 0 0) 0x7fd0bc1324d0 con 0x7fd0bc0b17d0 2024-03-29T11:48:35.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.638+0000 7fd0b97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:48:35.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.638+0000 7fd0b97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:35.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.638+0000 7fd0b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:35.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.638+0000 7fd0b97fa640 10 monclient(hunting): my global_id is 4252 2024-03-29T11:48:35.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.638+0000 7fd0b97fa640 1 -- 172.21.15.98:0/2740598096 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0a0004eb0 con 0x7fd0bc0b17d0 2024-03-29T11:48:35.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.638+0000 7fd0b97fa640 1 -- 172.21.15.98:0/2740598096 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4082349753 0 0) 0x7fd0bc1320c0 con 0x7fd0bc142690 2024-03-29T11:48:35.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.638+0000 7fd0b97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:48:35.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.638+0000 7fd0b97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:35.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.639+0000 7fd0b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:35.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.639+0000 7fd0b97fa640 10 monclient(hunting): my global_id is 4238 2024-03-29T11:48:35.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.639+0000 7fd0b97fa640 1 -- 172.21.15.98:0/2740598096 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0bc1324d0 con 0x7fd0bc142690 2024-03-29T11:48:35.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.639+0000 7fd0b97fa640 1 -- 172.21.15.98:0/2740598096 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1304184442 0 0) 0x7fd0a0004eb0 con 0x7fd0bc0b17d0 2024-03-29T11:48:35.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.639+0000 7fd0b97fa640 1 -- 172.21.15.98:0/2740598096 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd0bc1320c0 con 0x7fd0bc0b17d0 2024-03-29T11:48:35.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.639+0000 7fd0b97fa640 1 -- 172.21.15.98:0/2740598096 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fd0b0001bc0 con 0x7fd0bc0b17d0 2024-03-29T11:48:35.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.639+0000 7fd0b97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:35.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.639+0000 7fd0b97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:48:35.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.639+0000 7fd0b97fa640 10 monclient(hunting): dump: 2024-03-29T11:48:35.652 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:48:35.652 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:35.653 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:35.653 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:35.653 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:35.653 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:35.653 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:35.653 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:35.653 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:35.653 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:35.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.639+0000 7fd0b97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:48:35.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.640+0000 7fd0b97fa640 1 -- 172.21.15.98:0/2740598096 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2214637287 0 0) 0x7fd0bc1324d0 con 0x7fd0bc142690 2024-03-29T11:48:35.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.640+0000 7fd0b97fa640 1 -- 172.21.15.98:0/2740598096 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd0a0004eb0 con 0x7fd0bc142690 2024-03-29T11:48:35.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.640+0000 7fd0b97fa640 1 -- 172.21.15.98:0/2740598096 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fd0b4001b30 con 0x7fd0bc142690 2024-03-29T11:48:35.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.640+0000 7fd0b97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:35.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.640+0000 7fd0b97fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:48:35.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.640+0000 7fd0b97fa640 10 monclient(hunting): dump: 2024-03-29T11:48:35.654 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:48:35.654 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:35.654 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:35.654 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:35.654 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:35.654 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:35.654 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:35.654 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:35.654 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:35.654 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:35.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.640+0000 7fd0b97fa640 10 client.?.objecter ms_handle_connect 0x7fd0bc1422b0 2024-03-29T11:48:35.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.640+0000 7fd0b97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:35.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.641+0000 7fd0b97fa640 1 -- 172.21.15.98:0/2740598096 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1399171660 0 0) 0x7fd0a0004eb0 con 0x7fd0bc142690 2024-03-29T11:48:35.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.641+0000 7fd0b97fa640 1 -- 172.21.15.98:0/2740598096 >> v1:172.21.15.121:6789/0 conn(0x7fd0bc0b17d0 legacy=0x7fd0bc130080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:35.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.641+0000 7fd0b97fa640 1 -- 172.21.15.98:0/2740598096 >> v1:172.21.15.98:6789/0 conn(0x7fd0bc1422b0 legacy=0x7fd0bc130790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:35.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.641+0000 7fd0b97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:48:35.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.641+0000 7fd0b97fa640 1 monclient: found mon.c 2024-03-29T11:48:35.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.641+0000 7fd0b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:35.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.641+0000 7fd0b97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:48:35.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.641+0000 7fd0b97fa640 1 -- 172.21.15.98:0/2740598096 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0bc132250 con 0x7fd0bc142690 2024-03-29T11:48:35.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.641+0000 7fd0b97fa640 10 monclient: _finish_auth 0 2024-03-29T11:48:35.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.641+0000 7fd0b97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:48:35.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.641+0000 7fd0b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:35.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.641+0000 7fd0c2b97640 5 monclient: authenticate success, global_id 4238 2024-03-29T11:48:35.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.641+0000 7fd0c2b97640 10 monclient: _renew_subs 2024-03-29T11:48:35.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.641+0000 7fd0c2b97640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:48:35.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.641+0000 7fd0c2b97640 1 -- 172.21.15.98:0/2740598096 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0bc0abdb0 con 0x7fd0bc142690 2024-03-29T11:48:35.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.641+0000 7fd0c2b97640 10 client.4238.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:48:35.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.641+0000 7fd0c2b97640 10 monclient: _renew_subs 2024-03-29T11:48:35.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.641+0000 7fd0c2b97640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:48:35.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.641+0000 7fd0c2b97640 1 -- 172.21.15.98:0/2740598096 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0bc0ac370 con 0x7fd0bc142690 2024-03-29T11:48:35.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.641+0000 7fd0c2b97640 1 librados: init done 2024-03-29T11:48:35.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.641+0000 7fd0b97fa640 1 -- 172.21.15.98:0/2740598096 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd0b4001c90 con 0x7fd0bc142690 2024-03-29T11:48:35.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.641+0000 7fd0b97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:35.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.642+0000 7fd0b97fa640 1 -- 172.21.15.98:0/2740598096 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fd0b4003b10 con 0x7fd0bc142690 2024-03-29T11:48:35.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.642+0000 7fd0baffd640 4 set_mon_vals no callback set 2024-03-29T11:48:35.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.642+0000 7fd0b97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:35.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.642+0000 7fd0b97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:48:35.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.643+0000 7fd0b97fa640 10 monclient: dump: 2024-03-29T11:48:35.656 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:48:35.656 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:35.656 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:35.656 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:35.656 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:35.656 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:35.657 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:35.657 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:35.657 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:35.657 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:35.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.643+0000 7fd0c2b97640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:35.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.643+0000 7fd0c2b97640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:35.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.643+0000 7fd0c2b97640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:48:35.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.643+0000 7fd0c2b97640 1 -- 172.21.15.98:0/2740598096 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd09c005080 con 0x7fd0bc142690 2024-03-29T11:48:35.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.644+0000 7fd0b97fa640 1 -- 172.21.15.98:0/2740598096 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7fd0b4019ae0 con 0x7fd0bc142690 2024-03-29T11:48:35.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.645+0000 7fd0b97fa640 1 -- 172.21.15.98:0/2740598096 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7fd0b4083ae0 con 0x7fd0bc142690 2024-03-29T11:48:35.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.645+0000 7fd0b97fa640 10 client.4238.objecter ms_dispatch 0x7fd0bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:48:35.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.645+0000 7fd0b97fa640 3 client.4238.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:48:35.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.645+0000 7fd0b97fa640 3 client.4238.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:48:35.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.645+0000 7fd0b97fa640 20 client.4238.objecter dump_active .. 0 homeless 2024-03-29T11:48:35.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.645+0000 7fd0c0933640 10 monclient: get_auth_request con 0x7fd0a006de70 auth_method 0 2024-03-29T11:48:35.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.645+0000 7fd0b97fa640 10 client.4238.objecter ms_handle_connect 0x7fd0a006de70 2024-03-29T11:48:35.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.653+0000 7fd0b97fa640 1 -- 172.21.15.98:0/2740598096 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd0b404f3c0 con 0x7fd0bc142690 2024-03-29T11:48:35.660 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.653+0000 7fd0b97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:35.660 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.653+0000 7fd0b97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:48:35.953 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.947+0000 7fd0c2b97640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:35.953 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.947+0000 7fd0c2b97640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:35.953 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.947+0000 7fd0c2b97640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:48:35.953 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.947+0000 7fd0c2b97640 1 -- 172.21.15.98:0/2740598096 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd09c005330 con 0x7fd0bc142690 2024-03-29T11:48:35.954 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:48:35.954 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:48:35.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.948+0000 7fd0b97fa640 1 -- 172.21.15.98:0/2740598096 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7fd0b40512f0 con 0x7fd0bc142690 2024-03-29T11:48:35.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.948+0000 7fd0b97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:35.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.948+0000 7fd0b97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:48:35.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.951+0000 7fd0c2b97640 10 librados: watch_flush enter 2024-03-29T11:48:35.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.951+0000 7fd0c2b97640 10 librados: watch_flush exit 2024-03-29T11:48:35.958 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.952+0000 7fd0c2b97640 20 client.4238.objecter shutdown clearing up homeless session... 2024-03-29T11:48:35.958 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.952+0000 7fd0c2b97640 10 client.4238.objecter successfully canceled tick 2024-03-29T11:48:35.958 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.952+0000 7fd0c2b97640 1 -- 172.21.15.98:0/2740598096 >> v1:172.21.15.98:6816/4065186974 conn(0x7fd0a006de70 legacy=0x7fd0a0070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:35.958 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.952+0000 7fd0c2b97640 10 monclient: shutdown 2024-03-29T11:48:35.958 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.952+0000 7fd0c2b97640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:35.958 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.952+0000 7fd0c2b97640 1 -- 172.21.15.98:0/2740598096 >> v1:172.21.15.98:6790/0 conn(0x7fd0bc142690 legacy=0x7fd0bc1e1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:35.958 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.952+0000 7fd0c2b97640 1 -- 172.21.15.98:0/2740598096 shutdown_connections 2024-03-29T11:48:35.958 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.952+0000 7fd0c2b97640 1 -- 172.21.15.98:0/2740598096 >> 172.21.15.98:0/2740598096 conn(0x7fd0bc066a00 msgr2=0x7fd0bc13e1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:35.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.953+0000 7fd0c2b97640 1 -- 172.21.15.98:0/2740598096 shutdown_connections 2024-03-29T11:48:35.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.953+0000 7fd0c2b97640 1 -- 172.21.15.98:0/2740598096 wait complete. 2024-03-29T11:48:35.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:35.953+0000 7fd0c2b97640 1 librados: shutdown 2024-03-29T11:48:35.972 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:48:42.778 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:48:42.922 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.916+0000 7f6b46c6e640 10 monclient: get_monmap_and_config 2024-03-29T11:48:42.922 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.916+0000 7f6b46c6e640 10 monclient: build_initial_monmap 2024-03-29T11:48:42.922 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.916+0000 7f6b46c6e640 10 monclient: monmap: 2024-03-29T11:48:42.922 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:48:42.922 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:42.922 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:48:42.917531+0000 2024-03-29T11:48:42.923 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:48:42.917531+0000 2024-03-29T11:48:42.923 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:42.923 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:42.923 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:48:42.923 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:48:42.923 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:48:42.923 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:42.924 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.917+0000 7f6b46c6e640 1 Processor -- start 2024-03-29T11:48:42.924 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.917+0000 7f6b46c6e640 1 -- start start 2024-03-29T11:48:42.924 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.917+0000 7f6b46c6e640 10 monclient: init 2024-03-29T11:48:42.924 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.918+0000 7f6b46c6e640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:42.924 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.918+0000 7f6b46c6e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:48:42.925 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.918+0000 7f6b46c6e640 10 monclient(hunting): picked mon.noname-b con 0x7f6b400b0860 addr v1:172.21.15.98:6790/0 2024-03-29T11:48:42.925 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.920+0000 7f6b46c6e640 10 monclient(hunting): picked mon.noname-c con 0x7f6b4013ef30 addr v1:172.21.15.121:6789/0 2024-03-29T11:48:42.934 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.920+0000 7f6b46c6e640 10 monclient(hunting): picked mon.noname-a con 0x7f6b400af8e0 addr v1:172.21.15.98:6789/0 2024-03-29T11:48:42.934 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.920+0000 7f6b46c6e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b400b0c40 con 0x7f6b400af8e0 2024-03-29T11:48:42.934 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.920+0000 7f6b46c6e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b400aed10 con 0x7f6b4013ef30 2024-03-29T11:48:42.935 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.920+0000 7f6b46c6e640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b400afd50 con 0x7f6b400b0860 2024-03-29T11:48:42.935 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.920+0000 7f6b46c6e640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:42.935 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.920+0000 7f6b37fff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f6b4013ef30 0x7f6b400af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:57380/0 (socket says 172.21.15.98:57380) 2024-03-29T11:48:42.935 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.920+0000 7f6b37fff640 1 -- 172.21.15.98:0/662503596 learned_addr learned my addr 172.21.15.98:0/662503596 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:48:42.935 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.920+0000 7f6b46c6e640 10 monclient(hunting): authenticate will time out at 830.005462s 2024-03-29T11:48:42.935 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.922+0000 7f6b377fe640 1 -- 172.21.15.98:0/662503596 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1350569152 0 0) 0x7f6b400aed10 con 0x7f6b4013ef30 2024-03-29T11:48:42.935 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.922+0000 7f6b377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:48:42.935 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.922+0000 7f6b377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:42.935 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.922+0000 7f6b377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:42.935 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.922+0000 7f6b377fe640 10 monclient(hunting): my global_id is 4255 2024-03-29T11:48:42.935 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.922+0000 7f6b377fe640 1 -- 172.21.15.98:0/662503596 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b28007930 con 0x7f6b4013ef30 2024-03-29T11:48:42.935 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.922+0000 7f6b377fe640 1 -- 172.21.15.98:0/662503596 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2898137755 0 0) 0x7f6b28007930 con 0x7f6b4013ef30 2024-03-29T11:48:42.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.922+0000 7f6b377fe640 1 -- 172.21.15.98:0/662503596 >> v1:172.21.15.98:6790/0 conn(0x7f6b400b0860 legacy=0x7f6b4013e820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:48:42.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.923+0000 7f6b377fe640 1 -- 172.21.15.98:0/662503596 >> v1:172.21.15.98:6789/0 conn(0x7f6b400af8e0 legacy=0x7f6b40149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:48:42.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.923+0000 7f6b377fe640 10 monclient: _finish_hunting 0 2024-03-29T11:48:42.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.923+0000 7f6b377fe640 1 monclient: found mon.noname-c 2024-03-29T11:48:42.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.923+0000 7f6b377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:42.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.923+0000 7f6b377fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:48:42.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.923+0000 7f6b377fe640 1 -- 172.21.15.98:0/662503596 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b400ab1b0 con 0x7f6b4013ef30 2024-03-29T11:48:42.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.923+0000 7f6b377fe640 10 monclient: _finish_auth 0 2024-03-29T11:48:42.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.923+0000 7f6b377fe640 10 monclient: _check_auth_tickets 2024-03-29T11:48:42.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.923+0000 7f6b377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:42.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.923+0000 7f6b377fe640 1 -- 172.21.15.98:0/662503596 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f6b30001690 con 0x7f6b4013ef30 2024-03-29T11:48:42.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.923+0000 7f6b377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:42.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.923+0000 7f6b377fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:48:42.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.923+0000 7f6b377fe640 10 monclient: dump: 2024-03-29T11:48:42.937 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:48:42.937 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:42.937 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:42.937 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:42.937 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:42.937 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:42.937 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:42.937 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:42.937 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:42.937 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:42.937 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.923+0000 7f6b46c6e640 5 monclient: authenticate success, global_id 4255 2024-03-29T11:48:42.937 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.923+0000 7f6b46c6e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:48:42.938 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.923+0000 7f6b377fe640 1 -- 172.21.15.98:0/662503596 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b30001860 con 0x7f6b4013ef30 2024-03-29T11:48:42.938 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.923+0000 7f6b377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:42.938 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.923+0000 7f6b377fe640 1 -- 172.21.15.98:0/662503596 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f6b30001cb0 con 0x7f6b4013ef30 2024-03-29T11:48:42.938 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.923+0000 7f6b377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:42.938 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.923+0000 7f6b377fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:48:42.938 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.923+0000 7f6b377fe640 10 monclient: dump: 2024-03-29T11:48:42.938 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:48:42.938 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:42.938 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:42.938 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:42.938 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:42.938 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:42.939 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:42.939 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:42.939 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:42.939 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:42.939 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.923+0000 7f6b46c6e640 10 monclient: get_monmap_and_config success 2024-03-29T11:48:42.939 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.923+0000 7f6b46c6e640 4 set_mon_vals no callback set 2024-03-29T11:48:42.939 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.924+0000 7f6b46c6e640 10 monclient: shutdown 2024-03-29T11:48:42.939 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.924+0000 7f6b46c6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:42.939 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.924+0000 7f6b46c6e640 1 -- 172.21.15.98:0/662503596 >> v1:172.21.15.121:6789/0 conn(0x7f6b4013ef30 legacy=0x7f6b400af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:42.939 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.924+0000 7f6b46c6e640 1 -- 172.21.15.98:0/662503596 shutdown_connections 2024-03-29T11:48:42.939 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.924+0000 7f6b46c6e640 1 -- 172.21.15.98:0/662503596 >> 172.21.15.98:0/662503596 conn(0x7f6b4006b660 msgr2=0x7f6b4006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:42.939 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.924+0000 7f6b46c6e640 1 -- 172.21.15.98:0/662503596 shutdown_connections 2024-03-29T11:48:42.939 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.924+0000 7f6b46c6e640 1 -- 172.21.15.98:0/662503596 wait complete. 2024-03-29T11:48:42.940 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.925+0000 7f6b46c6e640 10 monclient: build_initial_monmap 2024-03-29T11:48:42.940 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.925+0000 7f6b46c6e640 10 monclient: monmap: 2024-03-29T11:48:42.940 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:48:42.940 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:48:42.940 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:48:42.940 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:48:42.940 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:42.940 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:42.940 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:48:42.940 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:48:42.940 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:48:42.940 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:42.941 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.926+0000 7f6b46c6e640 1 librados: starting msgr at 2024-03-29T11:48:42.941 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.926+0000 7f6b46c6e640 1 librados: starting objecter 2024-03-29T11:48:42.941 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.926+0000 7f6b46c6e640 1 Processor -- start 2024-03-29T11:48:42.941 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.926+0000 7f6b46c6e640 1 -- start start 2024-03-29T11:48:42.941 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.926+0000 7f6b46c6e640 1 librados: setting wanted keys 2024-03-29T11:48:42.941 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.926+0000 7f6b46c6e640 1 librados: calling monclient init 2024-03-29T11:48:42.941 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.926+0000 7f6b46c6e640 10 monclient: init 2024-03-29T11:48:42.941 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.927+0000 7f6b46c6e640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:42.941 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.927+0000 7f6b46c6e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:48:42.941 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.927+0000 7f6b46c6e640 10 monclient(hunting): picked mon.noname-c con 0x7f6b400af8e0 addr v1:172.21.15.98:6790/0 2024-03-29T11:48:42.941 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.927+0000 7f6b46c6e640 10 monclient(hunting): picked mon.noname-a con 0x7f6b400b0860 addr v1:172.21.15.98:6789/0 2024-03-29T11:48:42.941 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.927+0000 7f6b46c6e640 10 monclient(hunting): picked mon.noname-b con 0x7f6b4013ef30 addr v1:172.21.15.121:6789/0 2024-03-29T11:48:42.941 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.927+0000 7f6b46c6e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b40071ad0 con 0x7f6b400b0860 2024-03-29T11:48:42.942 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.927+0000 7f6b46c6e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b40136ce0 con 0x7f6b4013ef30 2024-03-29T11:48:42.942 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.927+0000 7f6b46c6e640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b4006fdb0 con 0x7f6b400af8e0 2024-03-29T11:48:42.942 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.927+0000 7f6b46c6e640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:42.942 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.927+0000 7f6b46c6e640 10 monclient(hunting): authenticate will time out at 830.012688s 2024-03-29T11:48:42.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.931+0000 7f6b44a0a640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f6b400af8e0 0x7f6b401365d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:46309/0 (socket says 172.21.15.98:46309) 2024-03-29T11:48:42.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.932+0000 7f6b44a0a640 1 -- 172.21.15.98:0/4174903217 learned_addr learned my addr 172.21.15.98:0/4174903217 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:48:42.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.932+0000 7f6b357fa640 10 client.?.objecter ms_handle_connect 0x7f6b400af8e0 2024-03-29T11:48:42.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.932+0000 7f6b357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:42.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.932+0000 7f6b357fa640 10 client.?.objecter ms_handle_connect 0x7f6b400b0860 2024-03-29T11:48:42.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.932+0000 7f6b357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:42.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.932+0000 7f6b357fa640 10 client.?.objecter ms_handle_connect 0x7f6b4013ef30 2024-03-29T11:48:42.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.932+0000 7f6b357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:42.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.932+0000 7f6b357fa640 1 -- 172.21.15.98:0/4174903217 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4185124692 0 0) 0x7f6b4006fdb0 con 0x7f6b400af8e0 2024-03-29T11:48:42.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.932+0000 7f6b357fa640 10 monclient(hunting): _negotiate 2024-03-29T11:48:42.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.932+0000 7f6b357fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:42.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.932+0000 7f6b357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:42.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.932+0000 7f6b357fa640 10 monclient(hunting): my global_id is 4241 2024-03-29T11:48:42.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.932+0000 7f6b357fa640 1 -- 172.21.15.98:0/4174903217 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b2c004eb0 con 0x7f6b400af8e0 2024-03-29T11:48:42.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.932+0000 7f6b357fa640 1 -- 172.21.15.98:0/4174903217 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1383908682 0 0) 0x7f6b40136ce0 con 0x7f6b4013ef30 2024-03-29T11:48:42.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.932+0000 7f6b357fa640 10 monclient(hunting): _negotiate 2024-03-29T11:48:42.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.932+0000 7f6b357fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:42.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.932+0000 7f6b357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:42.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.932+0000 7f6b357fa640 10 monclient(hunting): my global_id is 4258 2024-03-29T11:48:42.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.932+0000 7f6b357fa640 1 -- 172.21.15.98:0/4174903217 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b4006fdb0 con 0x7f6b4013ef30 2024-03-29T11:48:42.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.932+0000 7f6b357fa640 1 -- 172.21.15.98:0/4174903217 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3198779780 0 0) 0x7f6b40071ad0 con 0x7f6b400b0860 2024-03-29T11:48:42.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.932+0000 7f6b357fa640 10 monclient(hunting): _negotiate 2024-03-29T11:48:42.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.932+0000 7f6b357fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:42.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.932+0000 7f6b357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:42.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.932+0000 7f6b357fa640 10 monclient(hunting): my global_id is 4239 2024-03-29T11:48:42.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.932+0000 7f6b357fa640 1 -- 172.21.15.98:0/4174903217 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b40136ce0 con 0x7f6b400b0860 2024-03-29T11:48:42.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.932+0000 7f6b357fa640 1 -- 172.21.15.98:0/4174903217 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3070825577 0 0) 0x7f6b4006fdb0 con 0x7f6b4013ef30 2024-03-29T11:48:42.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.932+0000 7f6b357fa640 1 -- 172.21.15.98:0/4174903217 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b40071ad0 con 0x7f6b4013ef30 2024-03-29T11:48:42.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.932+0000 7f6b357fa640 1 -- 172.21.15.98:0/4174903217 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f6b38001a90 con 0x7f6b4013ef30 2024-03-29T11:48:42.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.932+0000 7f6b357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:42.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.932+0000 7f6b357fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:48:42.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.932+0000 7f6b357fa640 10 monclient(hunting): dump: 2024-03-29T11:48:42.951 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:48:42.951 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:42.951 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:42.951 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:42.952 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:42.952 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:42.952 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:42.952 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:42.952 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:42.952 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:42.952 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.933+0000 7f6b357fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:48:42.952 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.933+0000 7f6b357fa640 1 -- 172.21.15.98:0/4174903217 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3801821307 0 0) 0x7f6b2c004eb0 con 0x7f6b400af8e0 2024-03-29T11:48:42.952 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.933+0000 7f6b357fa640 1 -- 172.21.15.98:0/4174903217 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b4006fdb0 con 0x7f6b400af8e0 2024-03-29T11:48:42.952 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.933+0000 7f6b357fa640 1 -- 172.21.15.98:0/4174903217 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f6b3c001960 con 0x7f6b400af8e0 2024-03-29T11:48:42.952 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.933+0000 7f6b357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:42.952 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.933+0000 7f6b357fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:48:42.952 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.933+0000 7f6b357fa640 10 monclient(hunting): dump: 2024-03-29T11:48:42.953 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:48:42.953 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:42.953 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:42.953 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:42.953 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:42.953 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:42.953 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:42.953 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:42.953 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:42.953 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:42.953 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.933+0000 7f6b357fa640 1 -- 172.21.15.98:0/4174903217 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 48942448 0 0) 0x7f6b40071ad0 con 0x7f6b4013ef30 2024-03-29T11:48:42.953 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.933+0000 7f6b357fa640 1 -- 172.21.15.98:0/4174903217 >> v1:172.21.15.98:6790/0 conn(0x7f6b400af8e0 legacy=0x7f6b401365d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:42.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.933+0000 7f6b357fa640 1 -- 172.21.15.98:0/4174903217 >> v1:172.21.15.98:6789/0 conn(0x7f6b400b0860 legacy=0x7f6b40070790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:42.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.933+0000 7f6b357fa640 10 monclient: _finish_hunting 0 2024-03-29T11:48:42.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.933+0000 7f6b357fa640 1 monclient: found mon.b 2024-03-29T11:48:42.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.933+0000 7f6b357fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:42.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.933+0000 7f6b357fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:42.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.933+0000 7f6b357fa640 1 -- 172.21.15.98:0/4174903217 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b40070090 con 0x7f6b4013ef30 2024-03-29T11:48:42.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.933+0000 7f6b357fa640 10 monclient: _finish_auth 0 2024-03-29T11:48:42.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.933+0000 7f6b357fa640 10 monclient: _check_auth_tickets 2024-03-29T11:48:42.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.933+0000 7f6b357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:42.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.936+0000 7f6b46c6e640 5 monclient: authenticate success, global_id 4258 2024-03-29T11:48:42.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.936+0000 7f6b46c6e640 10 monclient: _renew_subs 2024-03-29T11:48:42.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.936+0000 7f6b46c6e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:42.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.936+0000 7f6b46c6e640 1 -- 172.21.15.98:0/4174903217 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b40070370 con 0x7f6b4013ef30 2024-03-29T11:48:42.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.936+0000 7f6b46c6e640 10 client.4258.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:48:42.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.936+0000 7f6b46c6e640 10 monclient: _renew_subs 2024-03-29T11:48:42.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.936+0000 7f6b46c6e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:42.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.936+0000 7f6b46c6e640 1 -- 172.21.15.98:0/4174903217 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b4006def0 con 0x7f6b4013ef30 2024-03-29T11:48:42.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.936+0000 7f6b46c6e640 1 librados: init done 2024-03-29T11:48:42.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.937+0000 7f6b26ffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:42.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.937+0000 7f6b26ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:42.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.937+0000 7f6b26ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:42.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.937+0000 7f6b26ffd640 1 -- 172.21.15.98:0/4174903217 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b400ab770 con 0x7f6b4013ef30 2024-03-29T11:48:42.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.940+0000 7f6b357fa640 1 -- 172.21.15.98:0/4174903217 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b38001f80 con 0x7f6b4013ef30 2024-03-29T11:48:42.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.940+0000 7f6b357fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:42.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.940+0000 7f6b357fa640 1 -- 172.21.15.98:0/4174903217 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f6b38003b10 con 0x7f6b4013ef30 2024-03-29T11:48:42.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.940+0000 7f6b357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:42.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.940+0000 7f6b357fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:48:42.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.940+0000 7f6b357fa640 10 monclient: dump: 2024-03-29T11:48:42.956 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:48:42.956 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:42.956 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:42.956 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:42.956 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:42.956 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:42.956 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:42.956 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:42.956 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:42.957 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:42.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.941+0000 7f6b367fc640 4 set_mon_vals no callback set 2024-03-29T11:48:42.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.941+0000 7f6b357fa640 1 -- 172.21.15.98:0/4174903217 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f6b38019ae0 con 0x7f6b4013ef30 2024-03-29T11:48:42.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.942+0000 7f6b357fa640 1 -- 172.21.15.98:0/4174903217 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f6b38083720 con 0x7f6b4013ef30 2024-03-29T11:48:42.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.942+0000 7f6b357fa640 10 client.4258.objecter ms_dispatch 0x7f6b40000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:48:42.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.942+0000 7f6b357fa640 3 client.4258.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:48:42.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.942+0000 7f6b357fa640 3 client.4258.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:48:42.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.942+0000 7f6b357fa640 20 client.4258.objecter dump_active .. 0 homeless 2024-03-29T11:48:42.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.943+0000 7f6b44a0a640 10 monclient: get_auth_request con 0x7f6b2c06e190 auth_method 0 2024-03-29T11:48:42.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.943+0000 7f6b357fa640 10 client.4258.objecter ms_handle_connect 0x7f6b2c06e190 2024-03-29T11:48:42.958 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.946+0000 7f6b357fa640 1 -- 172.21.15.98:0/4174903217 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6b3804f000 con 0x7f6b4013ef30 2024-03-29T11:48:42.958 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.946+0000 7f6b357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:42.958 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:42.946+0000 7f6b357fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:48:43.263 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:43.256+0000 7f6b26ffd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:43.263 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:43.256+0000 7f6b26ffd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:43.263 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:43.256+0000 7f6b26ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:48:43.263 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:43.256+0000 7f6b26ffd640 1 -- 172.21.15.98:0/4174903217 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b400ab770 con 0x7f6b4013ef30 2024-03-29T11:48:43.263 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:43.257+0000 7f6b357fa640 1 -- 172.21.15.98:0/4174903217 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f6b400ab770 con 0x7f6b4013ef30 2024-03-29T11:48:43.263 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:43.257+0000 7f6b357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:43.263 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:43.257+0000 7f6b357fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:48:43.264 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:48:43.264 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:48:43.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:43.260+0000 7f6b26ffd640 10 librados: watch_flush enter 2024-03-29T11:48:43.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:43.260+0000 7f6b26ffd640 10 librados: watch_flush exit 2024-03-29T11:48:43.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:43.261+0000 7f6b26ffd640 20 client.4258.objecter shutdown clearing up homeless session... 2024-03-29T11:48:43.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:43.261+0000 7f6b26ffd640 10 client.4258.objecter successfully canceled tick 2024-03-29T11:48:43.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:43.261+0000 7f6b26ffd640 1 -- 172.21.15.98:0/4174903217 >> v1:172.21.15.98:6816/4065186974 conn(0x7f6b2c06e190 legacy=0x7f6b2c070650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:43.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:43.261+0000 7f6b26ffd640 10 monclient: shutdown 2024-03-29T11:48:43.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:43.261+0000 7f6b26ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:43.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:43.261+0000 7f6b26ffd640 1 -- 172.21.15.98:0/4174903217 >> v1:172.21.15.121:6789/0 conn(0x7f6b4013ef30 legacy=0x7f6b40070ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:43.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:43.262+0000 7f6b26ffd640 1 -- 172.21.15.98:0/4174903217 shutdown_connections 2024-03-29T11:48:43.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:43.262+0000 7f6b26ffd640 1 -- 172.21.15.98:0/4174903217 >> 172.21.15.98:0/4174903217 conn(0x7f6b4006b660 msgr2=0x7f6b4006ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:43.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:43.262+0000 7f6b26ffd640 1 -- 172.21.15.98:0/4174903217 shutdown_connections 2024-03-29T11:48:43.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:43.262+0000 7f6b26ffd640 1 -- 172.21.15.98:0/4174903217 wait complete. 2024-03-29T11:48:43.269 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:43.262+0000 7f6b26ffd640 1 librados: shutdown 2024-03-29T11:48:43.283 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:48:50.090 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:48:50.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.227+0000 7fe0bd52b640 10 monclient: get_monmap_and_config 2024-03-29T11:48:50.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.227+0000 7fe0bd52b640 10 monclient: build_initial_monmap 2024-03-29T11:48:50.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.227+0000 7fe0bd52b640 10 monclient: monmap: 2024-03-29T11:48:50.234 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:48:50.234 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:50.234 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:48:50.229745+0000 2024-03-29T11:48:50.234 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:48:50.229745+0000 2024-03-29T11:48:50.234 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:50.235 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:50.235 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:48:50.235 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:48:50.235 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:48:50.235 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:50.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.228+0000 7fe0bd52b640 1 Processor -- start 2024-03-29T11:48:50.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.228+0000 7fe0bd52b640 1 -- start start 2024-03-29T11:48:50.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.228+0000 7fe0bd52b640 10 monclient: init 2024-03-29T11:48:50.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.229+0000 7fe0bd52b640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:50.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.229+0000 7fe0bd52b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:48:50.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.229+0000 7fe0bd52b640 10 monclient(hunting): picked mon.noname-b con 0x7fe0b81457f0 addr v1:172.21.15.98:6790/0 2024-03-29T11:48:50.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.229+0000 7fe0bd52b640 10 monclient(hunting): picked mon.noname-c con 0x7fe0b813f650 addr v1:172.21.15.121:6789/0 2024-03-29T11:48:50.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.231+0000 7fe0bd52b640 10 monclient(hunting): picked mon.noname-a con 0x7fe0b80a9d70 addr v1:172.21.15.98:6789/0 2024-03-29T11:48:50.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.231+0000 7fe0bd52b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0b80aaef0 con 0x7fe0b80a9d70 2024-03-29T11:48:50.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.231+0000 7fe0bd52b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0b8145bd0 con 0x7fe0b813f650 2024-03-29T11:48:50.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.231+0000 7fe0bd52b640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0b8145eb0 con 0x7fe0b81457f0 2024-03-29T11:48:50.239 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.231+0000 7fe0bd52b640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:50.239 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.231+0000 7fe0bd52b640 10 monclient(hunting): authenticate will time out at 837.317636s 2024-03-29T11:48:50.242 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.232+0000 7fe0b67fc640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fe0b813f650 0x7fe0b80b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:57598/0 (socket says 172.21.15.98:57598) 2024-03-29T11:48:50.242 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.232+0000 7fe0b67fc640 1 -- 172.21.15.98:0/707051592 learned_addr learned my addr 172.21.15.98:0/707051592 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:48:50.242 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b5ffb640 1 -- 172.21.15.98:0/707051592 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3288347528 0 0) 0x7fe0b8145eb0 con 0x7fe0b81457f0 2024-03-29T11:48:50.242 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:48:50.242 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:50.242 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:50.242 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b5ffb640 10 monclient(hunting): my global_id is 4244 2024-03-29T11:48:50.243 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b5ffb640 1 -- 172.21.15.98:0/707051592 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0a0003520 con 0x7fe0b81457f0 2024-03-29T11:48:50.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b5ffb640 1 -- 172.21.15.98:0/707051592 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 467767127 0 0) 0x7fe0b8145bd0 con 0x7fe0b813f650 2024-03-29T11:48:50.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:48:50.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:50.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:50.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b5ffb640 10 monclient(hunting): my global_id is 4261 2024-03-29T11:48:50.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b5ffb640 1 -- 172.21.15.98:0/707051592 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0b8145eb0 con 0x7fe0b813f650 2024-03-29T11:48:50.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b5ffb640 1 -- 172.21.15.98:0/707051592 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2150036239 0 0) 0x7fe0b80aaef0 con 0x7fe0b80a9d70 2024-03-29T11:48:50.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:48:50.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:50.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:50.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b5ffb640 10 monclient(hunting): my global_id is 4242 2024-03-29T11:48:50.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b5ffb640 1 -- 172.21.15.98:0/707051592 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0b8145bd0 con 0x7fe0b80a9d70 2024-03-29T11:48:50.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b5ffb640 1 -- 172.21.15.98:0/707051592 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1855592567 0 0) 0x7fe0b8145bd0 con 0x7fe0b80a9d70 2024-03-29T11:48:50.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b5ffb640 1 -- 172.21.15.98:0/707051592 >> v1:172.21.15.98:6790/0 conn(0x7fe0b81457f0 legacy=0x7fe0b813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:50.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b5ffb640 1 -- 172.21.15.98:0/707051592 >> v1:172.21.15.121:6789/0 conn(0x7fe0b813f650 legacy=0x7fe0b80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:50.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b67fc640 1 -- 172.21.15.98:0/707051592 >> v1:172.21.15.121:6789/0 conn(0x7fe0b813f650 legacy=0x7fe0b80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-03-29T11:48:50.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b67fc640 1 -- 172.21.15.98:0/707051592 >> v1:172.21.15.121:6789/0 conn(0x7fe0b813f650 legacy=0x7fe0b80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:48:50.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b67fc640 1 --1- 172.21.15.98:0/707051592 >> v1:172.21.15.121:6789/0 conn(0x7fe0b813f650 0x7fe0b80b2620 :-1 s=OPENED pgs=79642 cs=1 l=1).handle_message read tag failed 2024-03-29T11:48:50.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b67fc640 1 --1- 172.21.15.98:0/707051592 >> v1:172.21.15.121:6789/0 conn(0x7fe0b813f650 0x7fe0b80b2620 :-1 s=OPENED pgs=79642 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:48:50.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:48:50.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b5ffb640 1 monclient: found mon.noname-a 2024-03-29T11:48:50.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:50.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b5ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T11:48:50.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b5ffb640 1 -- 172.21.15.98:0/707051592 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0b8146340 con 0x7fe0b80a9d70 2024-03-29T11:48:50.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:48:50.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:48:50.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:50.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b5ffb640 10 monclient: ms_handle_reset stray mon v1:172.21.15.121:6789/0 2024-03-29T11:48:50.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b5ffb640 1 -- 172.21.15.98:0/707051592 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fe0a8001460 con 0x7fe0b80a9d70 2024-03-29T11:48:50.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:50.246 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:48:50.246 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b5ffb640 10 monclient: dump: 2024-03-29T11:48:50.246 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:48:50.246 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:50.246 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:50.246 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:50.246 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:50.246 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:50.246 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:50.246 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:50.246 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:50.246 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:50.246 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0bd52b640 5 monclient: authenticate success, global_id 4242 2024-03-29T11:48:50.246 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0bd52b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:48:50.246 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b5ffb640 1 -- 172.21.15.98:0/707051592 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0a80015c0 con 0x7fe0b80a9d70 2024-03-29T11:48:50.247 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:50.247 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b5ffb640 1 -- 172.21.15.98:0/707051592 <== mon.0 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fe0a80019d0 con 0x7fe0b80a9d70 2024-03-29T11:48:50.247 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:50.247 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:48:50.247 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0b5ffb640 10 monclient: dump: 2024-03-29T11:48:50.247 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:48:50.247 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:50.247 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:50.247 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:50.247 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:50.247 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:50.247 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:50.247 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:50.247 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:50.247 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:50.247 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0bd52b640 10 monclient: get_monmap_and_config success 2024-03-29T11:48:50.248 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.235+0000 7fe0bd52b640 4 set_mon_vals no callback set 2024-03-29T11:48:50.248 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.236+0000 7fe0bd52b640 10 monclient: shutdown 2024-03-29T11:48:50.248 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.236+0000 7fe0bd52b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:50.248 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.236+0000 7fe0bd52b640 1 -- 172.21.15.98:0/707051592 >> v1:172.21.15.98:6789/0 conn(0x7fe0b80a9d70 legacy=0x7fe0b814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:50.248 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.237+0000 7fe0bd52b640 1 -- 172.21.15.98:0/707051592 shutdown_connections 2024-03-29T11:48:50.248 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.237+0000 7fe0bd52b640 1 -- 172.21.15.98:0/707051592 >> 172.21.15.98:0/707051592 conn(0x7fe0b8066a00 msgr2=0x7fe0b8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:50.248 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.237+0000 7fe0bd52b640 1 -- 172.21.15.98:0/707051592 shutdown_connections 2024-03-29T11:48:50.249 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.237+0000 7fe0bd52b640 1 -- 172.21.15.98:0/707051592 wait complete. 2024-03-29T11:48:50.249 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.238+0000 7fe0bd52b640 10 monclient: build_initial_monmap 2024-03-29T11:48:50.249 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.238+0000 7fe0bd52b640 10 monclient: monmap: 2024-03-29T11:48:50.249 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:48:50.249 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:48:50.249 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:48:50.249 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:48:50.250 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:50.250 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:50.250 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:48:50.250 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:48:50.250 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:48:50.250 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:50.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.238+0000 7fe0bd52b640 1 librados: starting msgr at 2024-03-29T11:48:50.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.238+0000 7fe0bd52b640 1 librados: starting objecter 2024-03-29T11:48:50.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.239+0000 7fe0bd52b640 1 Processor -- start 2024-03-29T11:48:50.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.239+0000 7fe0bd52b640 1 -- start start 2024-03-29T11:48:50.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.239+0000 7fe0bd52b640 1 librados: setting wanted keys 2024-03-29T11:48:50.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.239+0000 7fe0bd52b640 1 librados: calling monclient init 2024-03-29T11:48:50.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.239+0000 7fe0bd52b640 10 monclient: init 2024-03-29T11:48:50.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.240+0000 7fe0bd52b640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:50.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.240+0000 7fe0bd52b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:48:50.251 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.240+0000 7fe0bd52b640 10 monclient(hunting): picked mon.noname-a con 0x7fe0b80a9d70 addr v1:172.21.15.98:6789/0 2024-03-29T11:48:50.251 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.240+0000 7fe0bd52b640 10 monclient(hunting): picked mon.noname-b con 0x7fe0b813f650 addr v1:172.21.15.121:6789/0 2024-03-29T11:48:50.251 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.240+0000 7fe0bd52b640 10 monclient(hunting): picked mon.noname-c con 0x7fe0b81457f0 addr v1:172.21.15.98:6790/0 2024-03-29T11:48:50.251 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.240+0000 7fe0bd52b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0b81452b0 con 0x7fe0b80a9d70 2024-03-29T11:48:50.251 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.240+0000 7fe0bd52b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0b81d0a50 con 0x7fe0b813f650 2024-03-29T11:48:50.251 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.240+0000 7fe0bd52b640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0b8135bd0 con 0x7fe0b81457f0 2024-03-29T11:48:50.251 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.240+0000 7fe0bd52b640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:50.251 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.240+0000 7fe0bd52b640 10 monclient(hunting): authenticate will time out at 837.326504s 2024-03-29T11:48:50.251 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.240+0000 7fe0b67fc640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fe0b813f650 0x7fe0b81354c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:57834/0 (socket says 172.21.15.98:57834) 2024-03-29T11:48:50.251 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.240+0000 7fe0b67fc640 1 -- 172.21.15.98:0/3013341907 learned_addr learned my addr 172.21.15.98:0/3013341907 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:48:50.252 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.243+0000 7fe09f7fe640 10 client.?.objecter ms_handle_connect 0x7fe0b813f650 2024-03-29T11:48:50.252 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.243+0000 7fe09f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:50.252 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.244+0000 7fe09f7fe640 10 client.?.objecter ms_handle_connect 0x7fe0b80a9d70 2024-03-29T11:48:50.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.244+0000 7fe09f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:50.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.244+0000 7fe09f7fe640 1 -- 172.21.15.98:0/3013341907 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1122461535 0 0) 0x7fe0b81452b0 con 0x7fe0b80a9d70 2024-03-29T11:48:50.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.244+0000 7fe09f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:48:50.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.244+0000 7fe09f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:50.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.244+0000 7fe09f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:50.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.244+0000 7fe09f7fe640 10 monclient(hunting): my global_id is 4245 2024-03-29T11:48:50.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.244+0000 7fe09f7fe640 1 -- 172.21.15.98:0/3013341907 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0a4004eb0 con 0x7fe0b80a9d70 2024-03-29T11:48:50.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.244+0000 7fe09f7fe640 10 client.?.objecter ms_handle_connect 0x7fe0b81457f0 2024-03-29T11:48:50.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.244+0000 7fe09f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:50.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.244+0000 7fe09f7fe640 1 -- 172.21.15.98:0/3013341907 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 672138083 0 0) 0x7fe0b8135bd0 con 0x7fe0b81457f0 2024-03-29T11:48:50.254 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.244+0000 7fe09f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:48:50.254 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.244+0000 7fe09f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:50.254 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.244+0000 7fe09f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:50.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.244+0000 7fe09f7fe640 10 monclient(hunting): my global_id is 4247 2024-03-29T11:48:50.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.244+0000 7fe09f7fe640 1 -- 172.21.15.98:0/3013341907 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0a40051d0 con 0x7fe0b81457f0 2024-03-29T11:48:50.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.244+0000 7fe09f7fe640 1 -- 172.21.15.98:0/3013341907 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 812575974 0 0) 0x7fe0b81d0a50 con 0x7fe0b813f650 2024-03-29T11:48:50.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.244+0000 7fe09f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:48:50.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.244+0000 7fe09f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:50.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.244+0000 7fe09f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:50.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.245+0000 7fe09f7fe640 10 monclient(hunting): my global_id is 4264 2024-03-29T11:48:50.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.245+0000 7fe09f7fe640 1 -- 172.21.15.98:0/3013341907 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0b8135bd0 con 0x7fe0b813f650 2024-03-29T11:48:50.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.245+0000 7fe09f7fe640 1 -- 172.21.15.98:0/3013341907 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2066305647 0 0) 0x7fe0a4004eb0 con 0x7fe0b80a9d70 2024-03-29T11:48:50.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.245+0000 7fe09f7fe640 1 -- 172.21.15.98:0/3013341907 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0a40057d0 con 0x7fe0b80a9d70 2024-03-29T11:48:50.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.245+0000 7fe09f7fe640 1 -- 172.21.15.98:0/3013341907 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fe0ac0018a0 con 0x7fe0b80a9d70 2024-03-29T11:48:50.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.245+0000 7fe09f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:50.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.245+0000 7fe09f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:48:50.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.245+0000 7fe09f7fe640 10 monclient(hunting): dump: 2024-03-29T11:48:50.255 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:48:50.255 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:50.255 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:50.256 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:50.256 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:50.256 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:50.256 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:50.256 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:50.256 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:50.256 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:50.256 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.245+0000 7fe09f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:48:50.256 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.245+0000 7fe09f7fe640 1 -- 172.21.15.98:0/3013341907 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2678820929 0 0) 0x7fe0a40057d0 con 0x7fe0b80a9d70 2024-03-29T11:48:50.256 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.245+0000 7fe09f7fe640 1 -- 172.21.15.98:0/3013341907 >> v1:172.21.15.98:6790/0 conn(0x7fe0b81457f0 legacy=0x7fe0b81d2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:50.256 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.245+0000 7fe09f7fe640 1 -- 172.21.15.98:0/3013341907 >> v1:172.21.15.121:6789/0 conn(0x7fe0b813f650 legacy=0x7fe0b81354c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:50.256 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.245+0000 7fe09f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:48:50.256 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.245+0000 7fe09f7fe640 1 monclient: found mon.a 2024-03-29T11:48:50.256 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.245+0000 7fe09f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:50.256 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.245+0000 7fe09f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:48:50.256 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.245+0000 7fe09f7fe640 1 -- 172.21.15.98:0/3013341907 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0b813db80 con 0x7fe0b80a9d70 2024-03-29T11:48:50.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.245+0000 7fe09f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:48:50.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.245+0000 7fe09f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:48:50.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.245+0000 7fe09f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:50.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.245+0000 7fe0bd52b640 5 monclient: authenticate success, global_id 4245 2024-03-29T11:48:50.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.245+0000 7fe0bd52b640 10 monclient: _renew_subs 2024-03-29T11:48:50.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.245+0000 7fe0bd52b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:48:50.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.245+0000 7fe0bd52b640 1 -- 172.21.15.98:0/3013341907 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0b8144780 con 0x7fe0b80a9d70 2024-03-29T11:48:50.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.245+0000 7fe0bd52b640 10 client.4245.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:48:50.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.245+0000 7fe0bd52b640 10 monclient: _renew_subs 2024-03-29T11:48:50.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.245+0000 7fe0bd52b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:48:50.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.245+0000 7fe0bd52b640 1 -- 172.21.15.98:0/3013341907 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0b8143320 con 0x7fe0b80a9d70 2024-03-29T11:48:50.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.245+0000 7fe0bd52b640 1 librados: init done 2024-03-29T11:48:50.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.246+0000 7fe09f7fe640 1 -- 172.21.15.98:0/3013341907 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0ac001d30 con 0x7fe0b80a9d70 2024-03-29T11:48:50.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.246+0000 7fe09f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:50.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.246+0000 7fe09f7fe640 1 -- 172.21.15.98:0/3013341907 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fe0ac0018a0 con 0x7fe0b80a9d70 2024-03-29T11:48:50.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.246+0000 7fe09f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:50.258 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.246+0000 7fe09f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:48:50.258 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.246+0000 7fe09f7fe640 10 monclient: dump: 2024-03-29T11:48:50.258 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:48:50.258 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:50.258 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:50.258 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:50.258 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:50.258 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:50.258 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:50.258 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:50.258 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:50.258 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:50.258 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.246+0000 7fe0b57fa640 4 set_mon_vals no callback set 2024-03-29T11:48:50.258 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.247+0000 7fe0bd52b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:50.259 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.247+0000 7fe0bd52b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:50.259 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.247+0000 7fe0bd52b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:48:50.259 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.247+0000 7fe0bd52b640 1 -- 172.21.15.98:0/3013341907 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe098005080 con 0x7fe0b80a9d70 2024-03-29T11:48:50.260 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.248+0000 7fe09f7fe640 1 -- 172.21.15.98:0/3013341907 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7fe0ac0194e0 con 0x7fe0b80a9d70 2024-03-29T11:48:50.260 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.250+0000 7fe09f7fe640 1 -- 172.21.15.98:0/3013341907 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7fe0ac0831a0 con 0x7fe0b80a9d70 2024-03-29T11:48:50.260 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.250+0000 7fe09f7fe640 10 client.4245.objecter ms_dispatch 0x7fe0b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:48:50.260 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.250+0000 7fe09f7fe640 3 client.4245.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:48:50.260 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.250+0000 7fe09f7fe640 3 client.4245.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:48:50.260 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.250+0000 7fe09f7fe640 20 client.4245.objecter dump_active .. 0 homeless 2024-03-29T11:48:50.260 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.250+0000 7fe0b67fc640 10 monclient: get_auth_request con 0x7fe0a406e600 auth_method 0 2024-03-29T11:48:50.260 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.250+0000 7fe09f7fe640 10 client.4245.objecter ms_handle_connect 0x7fe0a406e600 2024-03-29T11:48:50.265 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.258+0000 7fe09f7fe640 1 -- 172.21.15.98:0/3013341907 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe0ac04e9f0 con 0x7fe0b80a9d70 2024-03-29T11:48:50.265 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.258+0000 7fe09f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:50.265 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.258+0000 7fe09f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:48:50.594 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.587+0000 7fe0bd52b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:50.594 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.587+0000 7fe0bd52b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:50.594 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.587+0000 7fe0bd52b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:48:50.594 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.587+0000 7fe0bd52b640 1 -- 172.21.15.98:0/3013341907 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe098005330 con 0x7fe0b80a9d70 2024-03-29T11:48:50.595 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.588+0000 7fe09f7fe640 1 -- 172.21.15.98:0/3013341907 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7fe0ac050af0 con 0x7fe0b80a9d70 2024-03-29T11:48:50.595 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.588+0000 7fe09f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:50.595 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.588+0000 7fe09f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:48:50.595 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:48:50.595 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:48:50.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.592+0000 7fe0bd52b640 10 librados: watch_flush enter 2024-03-29T11:48:50.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.592+0000 7fe0bd52b640 10 librados: watch_flush exit 2024-03-29T11:48:50.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.592+0000 7fe0bd52b640 20 client.4245.objecter shutdown clearing up homeless session... 2024-03-29T11:48:50.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.592+0000 7fe0bd52b640 10 client.4245.objecter successfully canceled tick 2024-03-29T11:48:50.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.592+0000 7fe0bd52b640 1 -- 172.21.15.98:0/3013341907 >> v1:172.21.15.98:6816/4065186974 conn(0x7fe0a406e600 legacy=0x7fe0a4070ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:50.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.592+0000 7fe0bd52b640 10 monclient: shutdown 2024-03-29T11:48:50.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.592+0000 7fe0bd52b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:50.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.592+0000 7fe0bd52b640 1 -- 172.21.15.98:0/3013341907 >> v1:172.21.15.98:6789/0 conn(0x7fe0b80a9d70 legacy=0x7fe0b8144ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:50.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.593+0000 7fe0bd52b640 1 -- 172.21.15.98:0/3013341907 shutdown_connections 2024-03-29T11:48:50.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.593+0000 7fe0bd52b640 1 -- 172.21.15.98:0/3013341907 >> 172.21.15.98:0/3013341907 conn(0x7fe0b8066a00 msgr2=0x7fe0b813a5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:50.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.594+0000 7fe0bd52b640 1 -- 172.21.15.98:0/3013341907 shutdown_connections 2024-03-29T11:48:50.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.594+0000 7fe0bd52b640 1 -- 172.21.15.98:0/3013341907 wait complete. 2024-03-29T11:48:50.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:50.594+0000 7fe0bd52b640 1 librados: shutdown 2024-03-29T11:48:50.617 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:48:57.424 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:48:57.574 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.567+0000 7f530ca10640 10 monclient: get_monmap_and_config 2024-03-29T11:48:57.574 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.567+0000 7f530ca10640 10 monclient: build_initial_monmap 2024-03-29T11:48:57.574 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.567+0000 7f530ca10640 10 monclient: monmap: 2024-03-29T11:48:57.574 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:48:57.574 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:57.575 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:48:57.569702+0000 2024-03-29T11:48:57.575 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:48:57.569702+0000 2024-03-29T11:48:57.575 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:57.575 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:57.575 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:48:57.575 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:48:57.575 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:48:57.575 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:57.577 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.570+0000 7f530ca10640 1 Processor -- start 2024-03-29T11:48:57.577 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.570+0000 7f530ca10640 1 -- start start 2024-03-29T11:48:57.577 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.570+0000 7f530ca10640 10 monclient: init 2024-03-29T11:48:57.578 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.571+0000 7f530ca10640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:57.578 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.571+0000 7f530ca10640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:48:57.578 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.571+0000 7f530ca10640 10 monclient(hunting): picked mon.noname-b con 0x7f5300006e30 addr v1:172.21.15.98:6790/0 2024-03-29T11:48:57.578 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.571+0000 7f530ca10640 10 monclient(hunting): picked mon.noname-c con 0x7f5300035e00 addr v1:172.21.15.121:6789/0 2024-03-29T11:48:57.578 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.571+0000 7f530ca10640 10 monclient(hunting): picked mon.noname-a con 0x7f53000c9670 addr v1:172.21.15.98:6789/0 2024-03-29T11:48:57.578 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.571+0000 7f530ca10640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5300035970 con 0x7f53000c9670 2024-03-29T11:48:57.579 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.571+0000 7f530ca10640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5300036280 con 0x7f5300035e00 2024-03-29T11:48:57.579 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.571+0000 7f530ca10640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5300035640 con 0x7f5300006e30 2024-03-29T11:48:57.579 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.571+0000 7f530ca10640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:57.579 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.571+0000 7f530ca10640 10 monclient(hunting): authenticate will time out at 844.657442s 2024-03-29T11:48:57.582 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.575+0000 7f53077fe640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f5300006e30 0x7f53000cbcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:57095/0 (socket says 172.21.15.98:57095) 2024-03-29T11:48:57.582 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.575+0000 7f53077fe640 1 -- 172.21.15.98:0/2603325545 learned_addr learned my addr 172.21.15.98:0/2603325545 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:48:57.584 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.577+0000 7f53067fc640 1 -- 172.21.15.98:0/2603325545 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1052611920 0 0) 0x7f5300035640 con 0x7f5300006e30 2024-03-29T11:48:57.584 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.577+0000 7f53067fc640 10 monclient(hunting): _negotiate 2024-03-29T11:48:57.584 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.577+0000 7f53067fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:57.584 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.577+0000 7f53067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:57.584 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.577+0000 7f53067fc640 10 monclient(hunting): my global_id is 4250 2024-03-29T11:48:57.584 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.577+0000 7f53067fc640 1 -- 172.21.15.98:0/2603325545 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52f8003520 con 0x7f5300006e30 2024-03-29T11:48:57.585 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.578+0000 7f53067fc640 1 -- 172.21.15.98:0/2603325545 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2582445543 0 0) 0x7f52f8003520 con 0x7f5300006e30 2024-03-29T11:48:57.585 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.578+0000 7f53067fc640 1 -- 172.21.15.98:0/2603325545 >> v1:172.21.15.121:6789/0 conn(0x7f5300035e00 legacy=0x7f53000c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:57.585 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.578+0000 7f53067fc640 1 -- 172.21.15.98:0/2603325545 >> v1:172.21.15.98:6789/0 conn(0x7f53000c9670 legacy=0x7f53000d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:57.585 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.578+0000 7f53067fc640 10 monclient: _finish_hunting 0 2024-03-29T11:48:57.585 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.578+0000 7f53067fc640 1 monclient: found mon.noname-b 2024-03-29T11:48:57.585 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.578+0000 7f53067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:57.585 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.578+0000 7f53067fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:48:57.585 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.578+0000 7f53067fc640 1 -- 172.21.15.98:0/2603325545 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5300034e20 con 0x7f5300006e30 2024-03-29T11:48:57.585 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.578+0000 7f53067fc640 10 monclient: _finish_auth 0 2024-03-29T11:48:57.585 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.578+0000 7f53067fc640 10 monclient: _check_auth_tickets 2024-03-29T11:48:57.585 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.578+0000 7f53067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:57.586 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.578+0000 7f53067fc640 1 -- 172.21.15.98:0/2603325545 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f52fc001690 con 0x7f5300006e30 2024-03-29T11:48:57.586 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.578+0000 7f53067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:57.586 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.578+0000 7f53067fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:48:57.586 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.578+0000 7f53067fc640 10 monclient: dump: 2024-03-29T11:48:57.586 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:48:57.586 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:57.586 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:57.586 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:57.586 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:57.586 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:57.586 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:57.586 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:57.586 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:57.586 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:57.587 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.578+0000 7f530ca10640 5 monclient: authenticate success, global_id 4250 2024-03-29T11:48:57.587 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.578+0000 7f530ca10640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:48:57.587 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.578+0000 7f53067fc640 1 -- 172.21.15.98:0/2603325545 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52fc001860 con 0x7f5300006e30 2024-03-29T11:48:57.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.578+0000 7f53067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:57.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.578+0000 7f53067fc640 1 -- 172.21.15.98:0/2603325545 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f52fc001c70 con 0x7f5300006e30 2024-03-29T11:48:57.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.578+0000 7f53067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:57.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.578+0000 7f53067fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:48:57.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.578+0000 7f53067fc640 10 monclient: dump: 2024-03-29T11:48:57.588 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:48:57.588 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:57.588 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:57.588 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:57.588 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:57.588 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:57.588 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:57.588 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:57.589 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:57.589 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:57.589 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.578+0000 7f530ca10640 10 monclient: get_monmap_and_config success 2024-03-29T11:48:57.589 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.578+0000 7f530ca10640 4 set_mon_vals no callback set 2024-03-29T11:48:57.589 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.579+0000 7f530ca10640 10 monclient: shutdown 2024-03-29T11:48:57.589 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.579+0000 7f530ca10640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:57.589 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.579+0000 7f530ca10640 1 -- 172.21.15.98:0/2603325545 >> v1:172.21.15.98:6790/0 conn(0x7f5300006e30 legacy=0x7f53000cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:57.589 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.581+0000 7f530ca10640 1 -- 172.21.15.98:0/2603325545 shutdown_connections 2024-03-29T11:48:57.589 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.582+0000 7f530ca10640 1 -- 172.21.15.98:0/2603325545 >> 172.21.15.98:0/2603325545 conn(0x7f5300005c00 msgr2=0x7f5300006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:57.591 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.584+0000 7f530ca10640 1 -- 172.21.15.98:0/2603325545 shutdown_connections 2024-03-29T11:48:57.595 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.588+0000 7f530ca10640 1 -- 172.21.15.98:0/2603325545 wait complete. 2024-03-29T11:48:57.596 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.590+0000 7f530ca10640 10 monclient: build_initial_monmap 2024-03-29T11:48:57.596 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.590+0000 7f530ca10640 10 monclient: monmap: 2024-03-29T11:48:57.596 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:48:57.596 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:48:57.597 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:48:57.597 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:48:57.597 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:48:57.597 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:57.597 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:48:57.597 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:48:57.597 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:48:57.597 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:57.598 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.590+0000 7f530ca10640 1 librados: starting msgr at 2024-03-29T11:48:57.598 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.590+0000 7f530ca10640 1 librados: starting objecter 2024-03-29T11:48:57.598 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.591+0000 7f530ca10640 1 Processor -- start 2024-03-29T11:48:57.598 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.591+0000 7f530ca10640 1 -- start start 2024-03-29T11:48:57.598 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.591+0000 7f530ca10640 1 librados: setting wanted keys 2024-03-29T11:48:57.598 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.591+0000 7f530ca10640 1 librados: calling monclient init 2024-03-29T11:48:57.598 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.591+0000 7f530ca10640 10 monclient: init 2024-03-29T11:48:57.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.592+0000 7f530ca10640 10 monclient: _reopen_session rank -1 2024-03-29T11:48:57.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.592+0000 7f530ca10640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:48:57.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.592+0000 7f530ca10640 10 monclient(hunting): picked mon.noname-c con 0x7f5300006e30 addr v1:172.21.15.98:6790/0 2024-03-29T11:48:57.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.592+0000 7f530ca10640 10 monclient(hunting): picked mon.noname-a con 0x7f5300035e00 addr v1:172.21.15.98:6789/0 2024-03-29T11:48:57.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.592+0000 7f530ca10640 10 monclient(hunting): picked mon.noname-b con 0x7f53000c9670 addr v1:172.21.15.121:6789/0 2024-03-29T11:48:57.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.592+0000 7f530ca10640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5300033a20 con 0x7f5300035e00 2024-03-29T11:48:57.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.592+0000 7f530ca10640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5300032410 con 0x7f53000c9670 2024-03-29T11:48:57.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.592+0000 7f530ca10640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5300033c50 con 0x7f5300006e30 2024-03-29T11:48:57.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.592+0000 7f530ca10640 10 monclient(hunting): _renew_subs 2024-03-29T11:48:57.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.592+0000 7f530ca10640 10 monclient(hunting): authenticate will time out at 844.678778s 2024-03-29T11:48:57.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.596+0000 7f53077fe640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f5300006e30 0x7f5300031b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:57223/0 (socket says 172.21.15.98:57223) 2024-03-29T11:48:57.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.596+0000 7f53077fe640 1 -- 172.21.15.98:0/1400658098 learned_addr learned my addr 172.21.15.98:0/1400658098 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:48:57.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.596+0000 7f52f3fff640 10 client.?.objecter ms_handle_connect 0x7f5300035e00 2024-03-29T11:48:57.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.596+0000 7f52f3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:57.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.596+0000 7f52f3fff640 10 client.?.objecter ms_handle_connect 0x7f5300006e30 2024-03-29T11:48:57.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.596+0000 7f52f3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:57.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.596+0000 7f52f3fff640 1 -- 172.21.15.98:0/1400658098 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 451290786 0 0) 0x7f5300033c50 con 0x7f5300006e30 2024-03-29T11:48:57.605 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.598+0000 7f52f3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:48:57.605 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.598+0000 7f52f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:57.605 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.598+0000 7f52f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:57.605 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.598+0000 7f52f3fff640 10 monclient(hunting): my global_id is 4253 2024-03-29T11:48:57.606 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.598+0000 7f52f3fff640 1 -- 172.21.15.98:0/1400658098 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52ec004eb0 con 0x7f5300006e30 2024-03-29T11:48:57.606 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.598+0000 7f52f3fff640 1 -- 172.21.15.98:0/1400658098 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1682259700 0 0) 0x7f5300033a20 con 0x7f5300035e00 2024-03-29T11:48:57.606 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.598+0000 7f52f3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:48:57.606 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.598+0000 7f52f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:57.606 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.598+0000 7f52f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:57.606 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.598+0000 7f52f3fff640 10 monclient(hunting): my global_id is 4251 2024-03-29T11:48:57.606 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.598+0000 7f52f3fff640 1 -- 172.21.15.98:0/1400658098 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5300033c50 con 0x7f5300035e00 2024-03-29T11:48:57.606 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.599+0000 7f52f3fff640 10 client.?.objecter ms_handle_connect 0x7f53000c9670 2024-03-29T11:48:57.606 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.599+0000 7f52f3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:48:57.609 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.600+0000 7f52f3fff640 1 -- 172.21.15.98:0/1400658098 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4081819644 0 0) 0x7f5300033c50 con 0x7f5300035e00 2024-03-29T11:48:57.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.600+0000 7f52f3fff640 1 -- 172.21.15.98:0/1400658098 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5300033a20 con 0x7f5300035e00 2024-03-29T11:48:57.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.600+0000 7f52f3fff640 1 -- 172.21.15.98:0/1400658098 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f530804fdd0 con 0x7f5300035e00 2024-03-29T11:48:57.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.600+0000 7f52f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:57.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.600+0000 7f52f3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:48:57.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.600+0000 7f52f3fff640 10 monclient(hunting): dump: 2024-03-29T11:48:57.610 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:48:57.610 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:57.610 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:57.610 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:57.610 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:57.610 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:57.610 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:57.610 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:57.611 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:57.611 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:57.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.600+0000 7f52f3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:48:57.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.600+0000 7f52f3fff640 1 -- 172.21.15.98:0/1400658098 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3299005037 0 0) 0x7f5300032410 con 0x7f53000c9670 2024-03-29T11:48:57.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.600+0000 7f52f3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:48:57.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.600+0000 7f52f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:48:57.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.600+0000 7f52f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:48:57.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.600+0000 7f52f3fff640 10 monclient(hunting): my global_id is 4270 2024-03-29T11:48:57.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.600+0000 7f52f3fff640 1 -- 172.21.15.98:0/1400658098 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5300033c50 con 0x7f53000c9670 2024-03-29T11:48:57.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.600+0000 7f52f3fff640 1 -- 172.21.15.98:0/1400658098 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1653145644 0 0) 0x7f52ec004eb0 con 0x7f5300006e30 2024-03-29T11:48:57.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.600+0000 7f52f3fff640 1 -- 172.21.15.98:0/1400658098 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5300032410 con 0x7f5300006e30 2024-03-29T11:48:57.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.600+0000 7f52f3fff640 1 -- 172.21.15.98:0/1400658098 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f52fc001b70 con 0x7f5300006e30 2024-03-29T11:48:57.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.600+0000 7f52f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:57.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.600+0000 7f52f3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:48:57.612 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.600+0000 7f52f3fff640 10 monclient(hunting): dump: 2024-03-29T11:48:57.612 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:48:57.612 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:57.612 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:57.612 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:57.612 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:57.612 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:57.612 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:57.612 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:57.612 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:57.612 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:57.612 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.600+0000 7f52f3fff640 1 -- 172.21.15.98:0/1400658098 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2893361933 0 0) 0x7f5300033a20 con 0x7f5300035e00 2024-03-29T11:48:57.612 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.600+0000 7f52f3fff640 1 -- 172.21.15.98:0/1400658098 >> v1:172.21.15.98:6790/0 conn(0x7f5300006e30 legacy=0x7f5300031b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:57.612 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.601+0000 7f52f3fff640 1 -- 172.21.15.98:0/1400658098 >> v1:172.21.15.121:6789/0 conn(0x7f53000c9670 legacy=0x7f53000bfd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:57.612 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.601+0000 7f52f3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:48:57.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.601+0000 7f52f3fff640 1 monclient: found mon.a 2024-03-29T11:48:57.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.601+0000 7f52f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:48:57.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.601+0000 7f52f3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:48:57.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.601+0000 7f52f3fff640 1 -- 172.21.15.98:0/1400658098 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5300032660 con 0x7f5300035e00 2024-03-29T11:48:57.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.601+0000 7f52f3fff640 10 monclient: _finish_auth 0 2024-03-29T11:48:57.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.601+0000 7f52f3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:48:57.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.601+0000 7f52f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:48:57.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.601+0000 7f530ca10640 5 monclient: authenticate success, global_id 4251 2024-03-29T11:48:57.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.601+0000 7f530ca10640 10 monclient: _renew_subs 2024-03-29T11:48:57.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.601+0000 7f530ca10640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:48:57.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.601+0000 7f530ca10640 1 -- 172.21.15.98:0/1400658098 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53000cb4f0 con 0x7f5300035e00 2024-03-29T11:48:57.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.601+0000 7f530ca10640 10 client.4251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:48:57.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.601+0000 7f530ca10640 10 monclient: _renew_subs 2024-03-29T11:48:57.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.601+0000 7f530ca10640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:48:57.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.601+0000 7f530ca10640 1 -- 172.21.15.98:0/1400658098 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f530002dd20 con 0x7f5300035e00 2024-03-29T11:48:57.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.601+0000 7f530ca10640 1 librados: init done 2024-03-29T11:48:57.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.601+0000 7f52f3fff640 1 -- 172.21.15.98:0/1400658098 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f530804f400 con 0x7f5300035e00 2024-03-29T11:48:57.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.601+0000 7f52f3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:48:57.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.601+0000 7f52f3fff640 1 -- 172.21.15.98:0/1400658098 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f5308075f10 con 0x7f5300035e00 2024-03-29T11:48:57.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.601+0000 7f52f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:48:57.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.601+0000 7f52f3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:48:57.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.601+0000 7f52f3fff640 10 monclient: dump: 2024-03-29T11:48:57.614 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:48:57.614 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:48:57.614 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:57.614 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:48:57.614 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:48:57.614 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:48:57.614 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:48:57.615 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:48:57.615 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:48:57.615 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:48:57.615 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.602+0000 7f5305ffb640 4 set_mon_vals no callback set 2024-03-29T11:48:57.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.604+0000 7f52f3fff640 1 -- 172.21.15.98:0/1400658098 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f5308067bf0 con 0x7f5300035e00 2024-03-29T11:48:57.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.604+0000 7f52f3fff640 1 -- 172.21.15.98:0/1400658098 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f53080fe350 con 0x7f5300035e00 2024-03-29T11:48:57.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.604+0000 7f52f3fff640 10 client.4251.objecter ms_dispatch 0x7f5308000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:48:57.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.604+0000 7f52f3fff640 3 client.4251.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:48:57.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.605+0000 7f52f3fff640 3 client.4251.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:48:57.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.605+0000 7f53077fe640 10 monclient: get_auth_request con 0x7f52ec06de50 auth_method 0 2024-03-29T11:48:57.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.605+0000 7f52f3fff640 20 client.4251.objecter dump_active .. 0 homeless 2024-03-29T11:48:57.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.606+0000 7f52f3fff640 10 client.4251.objecter ms_handle_connect 0x7f52ec06de50 2024-03-29T11:48:57.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.606+0000 7f530ca10640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:57.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.606+0000 7f530ca10640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:57.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.606+0000 7f530ca10640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:48:57.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.606+0000 7f530ca10640 1 -- 172.21.15.98:0/1400658098 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52e8005080 con 0x7f5300035e00 2024-03-29T11:48:57.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.617+0000 7f52f3fff640 1 -- 172.21.15.98:0/1400658098 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f53080c9c30 con 0x7f5300035e00 2024-03-29T11:48:57.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.617+0000 7f52f3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:48:57.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.617+0000 7f52f3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:48:57.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.942+0000 7f530ca10640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:57.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.942+0000 7f530ca10640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:57.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.942+0000 7f530ca10640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:48:57.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.942+0000 7f530ca10640 1 -- 172.21.15.98:0/1400658098 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52e8005330 con 0x7f5300035e00 2024-03-29T11:48:57.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.944+0000 7f52f3fff640 1 -- 172.21.15.98:0/1400658098 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f53080cbb60 con 0x7f5300035e00 2024-03-29T11:48:57.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.944+0000 7f52f3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:48:57.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.944+0000 7f52f3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:48:57.952 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:48:57.952 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:48:57.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.947+0000 7f530ca10640 10 librados: watch_flush enter 2024-03-29T11:48:57.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.947+0000 7f530ca10640 10 librados: watch_flush exit 2024-03-29T11:48:57.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.947+0000 7f530ca10640 20 client.4251.objecter shutdown clearing up homeless session... 2024-03-29T11:48:57.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.947+0000 7f530ca10640 10 client.4251.objecter successfully canceled tick 2024-03-29T11:48:57.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.947+0000 7f530ca10640 1 -- 172.21.15.98:0/1400658098 >> v1:172.21.15.98:6816/4065186974 conn(0x7f52ec06de50 legacy=0x7f52ec070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:57.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.947+0000 7f530ca10640 10 monclient: shutdown 2024-03-29T11:48:57.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.947+0000 7f530ca10640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:48:57.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.947+0000 7f530ca10640 1 -- 172.21.15.98:0/1400658098 >> v1:172.21.15.98:6789/0 conn(0x7f5300035e00 legacy=0x7f530002d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:48:57.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.948+0000 7f530ca10640 1 -- 172.21.15.98:0/1400658098 shutdown_connections 2024-03-29T11:48:57.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.948+0000 7f530ca10640 1 -- 172.21.15.98:0/1400658098 >> 172.21.15.98:0/1400658098 conn(0x7f5300005c00 msgr2=0x7f53000c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:48:57.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.948+0000 7f530ca10640 1 -- 172.21.15.98:0/1400658098 shutdown_connections 2024-03-29T11:48:57.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.948+0000 7f530ca10640 1 -- 172.21.15.98:0/1400658098 wait complete. 2024-03-29T11:48:57.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:48:57.948+0000 7f530ca10640 1 librados: shutdown 2024-03-29T11:48:57.980 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:49:04.787 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:49:04.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.930+0000 7fde803d0640 10 monclient: get_monmap_and_config 2024-03-29T11:49:04.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.930+0000 7fde803d0640 10 monclient: build_initial_monmap 2024-03-29T11:49:04.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.930+0000 7fde803d0640 10 monclient: monmap: 2024-03-29T11:49:04.936 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:49:04.936 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:49:04.936 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:49:04.931064+0000 2024-03-29T11:49:04.936 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:49:04.931064+0000 2024-03-29T11:49:04.936 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:49:04.936 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:04.936 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:49:04.936 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:49:04.936 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:49:04.936 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:04.937 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.931+0000 7fde803d0640 1 Processor -- start 2024-03-29T11:49:04.942 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.931+0000 7fde803d0640 1 -- start start 2024-03-29T11:49:04.942 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.931+0000 7fde803d0640 10 monclient: init 2024-03-29T11:49:04.942 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.932+0000 7fde803d0640 10 monclient: _reopen_session rank -1 2024-03-29T11:49:04.942 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.932+0000 7fde803d0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:49:04.942 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.932+0000 7fde803d0640 10 monclient(hunting): picked mon.noname-c con 0x7fde70006e30 addr v1:172.21.15.121:6789/0 2024-03-29T11:49:04.942 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.932+0000 7fde803d0640 10 monclient(hunting): picked mon.noname-a con 0x7fde70035e00 addr v1:172.21.15.98:6789/0 2024-03-29T11:49:04.942 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.932+0000 7fde803d0640 10 monclient(hunting): picked mon.noname-b con 0x7fde700c9670 addr v1:172.21.15.98:6790/0 2024-03-29T11:49:04.942 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.932+0000 7fde803d0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde70035970 con 0x7fde70035e00 2024-03-29T11:49:04.943 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.932+0000 7fde803d0640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde70036280 con 0x7fde70006e30 2024-03-29T11:49:04.943 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.932+0000 7fde803d0640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde70035640 con 0x7fde700c9670 2024-03-29T11:49:04.943 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.932+0000 7fde803d0640 10 monclient(hunting): _renew_subs 2024-03-29T11:49:04.943 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.932+0000 7fde803d0640 10 monclient(hunting): authenticate will time out at 852.017787s 2024-03-29T11:49:04.943 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.937+0000 7fde7d96b640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fde70035e00 0x7fde700c8f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:50463/0 (socket says 172.21.15.98:50463) 2024-03-29T11:49:04.943 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.938+0000 7fde7d96b640 1 -- 172.21.15.98:0/1033747833 learned_addr learned my addr 172.21.15.98:0/1033747833 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:49:04.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.938+0000 7fde7d16a640 1 -- 172.21.15.98:0/1033747833 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 712953848 0 0) 0x7fde70036280 con 0x7fde70006e30 2024-03-29T11:49:04.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.939+0000 7fde7d16a640 10 monclient(hunting): _negotiate 2024-03-29T11:49:04.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.939+0000 7fde7d16a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:04.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.939+0000 7fde7d16a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:04.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.939+0000 7fde7d16a640 10 monclient(hunting): my global_id is 4273 2024-03-29T11:49:04.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.939+0000 7fde7d16a640 1 -- 172.21.15.98:0/1033747833 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde64003520 con 0x7fde70006e30 2024-03-29T11:49:04.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.939+0000 7fde7d16a640 1 -- 172.21.15.98:0/1033747833 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 925374701 0 0) 0x7fde70035970 con 0x7fde70035e00 2024-03-29T11:49:04.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.939+0000 7fde7d16a640 10 monclient(hunting): _negotiate 2024-03-29T11:49:04.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.939+0000 7fde7d16a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:04.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.939+0000 7fde7d16a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:04.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.939+0000 7fde7d16a640 10 monclient(hunting): my global_id is 4254 2024-03-29T11:49:04.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.939+0000 7fde7d16a640 1 -- 172.21.15.98:0/1033747833 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde70036280 con 0x7fde70035e00 2024-03-29T11:49:04.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.939+0000 7fde7d16a640 1 -- 172.21.15.98:0/1033747833 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4115960713 0 0) 0x7fde64003520 con 0x7fde70006e30 2024-03-29T11:49:04.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.940+0000 7fde7d16a640 1 -- 172.21.15.98:0/1033747833 >> v1:172.21.15.98:6790/0 conn(0x7fde700c9670 legacy=0x7fde700d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:49:04.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.940+0000 7fde7d16a640 1 -- 172.21.15.98:0/1033747833 >> v1:172.21.15.98:6789/0 conn(0x7fde70035e00 legacy=0x7fde700c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:04.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.940+0000 7fde7d16a640 10 monclient: _finish_hunting 0 2024-03-29T11:49:04.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.940+0000 7fde7d16a640 1 monclient: found mon.noname-c 2024-03-29T11:49:04.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.940+0000 7fde7d16a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:49:04.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.940+0000 7fde7d16a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:49:04.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.940+0000 7fde7d16a640 1 -- 172.21.15.98:0/1033747833 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde70034e20 con 0x7fde70006e30 2024-03-29T11:49:04.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.940+0000 7fde7d16a640 10 monclient: _finish_auth 0 2024-03-29T11:49:04.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.940+0000 7fde7d16a640 10 monclient: _check_auth_tickets 2024-03-29T11:49:04.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.940+0000 7fde7d16a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:49:04.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.940+0000 7fde803d0640 5 monclient: authenticate success, global_id 4273 2024-03-29T11:49:04.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.940+0000 7fde803d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:49:04.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.940+0000 7fde7d16a640 1 -- 172.21.15.98:0/1033747833 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fde7804f730 con 0x7fde70006e30 2024-03-29T11:49:04.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.940+0000 7fde7d16a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:04.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.940+0000 7fde7d16a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:49:04.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.940+0000 7fde7d16a640 10 monclient: dump: 2024-03-29T11:49:04.949 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:49:04.949 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:49:04.949 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:04.949 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:04.949 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:04.949 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:04.949 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:49:04.949 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:49:04.949 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:49:04.949 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:04.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.941+0000 7fde803d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:49:04.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.941+0000 7fde7d16a640 1 -- 172.21.15.98:0/1033747833 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fde78075f10 con 0x7fde70006e30 2024-03-29T11:49:04.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.941+0000 7fde7d16a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:49:04.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.941+0000 7fde803d0640 10 monclient: get_monmap_and_config success 2024-03-29T11:49:04.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.941+0000 7fde803d0640 4 set_mon_vals no callback set 2024-03-29T11:49:04.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.942+0000 7fde803d0640 10 monclient: shutdown 2024-03-29T11:49:04.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.942+0000 7fde803d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:49:04.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.942+0000 7fde803d0640 1 -- 172.21.15.98:0/1033747833 >> v1:172.21.15.121:6789/0 conn(0x7fde70006e30 legacy=0x7fde700cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:04.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.942+0000 7fde803d0640 1 -- 172.21.15.98:0/1033747833 shutdown_connections 2024-03-29T11:49:04.952 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.942+0000 7fde803d0640 1 -- 172.21.15.98:0/1033747833 >> 172.21.15.98:0/1033747833 conn(0x7fde70005c00 msgr2=0x7fde70006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:49:04.952 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.942+0000 7fde803d0640 1 -- 172.21.15.98:0/1033747833 shutdown_connections 2024-03-29T11:49:04.952 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.942+0000 7fde803d0640 1 -- 172.21.15.98:0/1033747833 wait complete. 2024-03-29T11:49:04.952 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.943+0000 7fde803d0640 10 monclient: build_initial_monmap 2024-03-29T11:49:04.952 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.943+0000 7fde803d0640 10 monclient: monmap: 2024-03-29T11:49:04.952 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:49:04.952 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:49:04.952 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:49:04.952 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:49:04.952 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:49:04.952 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:04.952 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:49:04.952 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:49:04.952 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:49:04.952 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:04.952 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.944+0000 7fde803d0640 1 librados: starting msgr at 2024-03-29T11:49:04.953 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.944+0000 7fde803d0640 1 librados: starting objecter 2024-03-29T11:49:04.953 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.944+0000 7fde803d0640 1 Processor -- start 2024-03-29T11:49:04.953 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.944+0000 7fde803d0640 1 -- start start 2024-03-29T11:49:04.953 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.944+0000 7fde803d0640 1 librados: setting wanted keys 2024-03-29T11:49:04.953 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.944+0000 7fde803d0640 1 librados: calling monclient init 2024-03-29T11:49:04.953 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.944+0000 7fde803d0640 10 monclient: init 2024-03-29T11:49:04.953 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.945+0000 7fde803d0640 10 monclient: _reopen_session rank -1 2024-03-29T11:49:04.953 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.945+0000 7fde803d0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:49:04.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.945+0000 7fde803d0640 10 monclient(hunting): picked mon.noname-a con 0x7fde70006e30 addr v1:172.21.15.98:6789/0 2024-03-29T11:49:04.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.945+0000 7fde803d0640 10 monclient(hunting): picked mon.noname-c con 0x7fde70035e00 addr v1:172.21.15.98:6790/0 2024-03-29T11:49:04.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.945+0000 7fde803d0640 10 monclient(hunting): picked mon.noname-b con 0x7fde700c9670 addr v1:172.21.15.121:6789/0 2024-03-29T11:49:04.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.945+0000 7fde803d0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde70033980 con 0x7fde70006e30 2024-03-29T11:49:04.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.945+0000 7fde803d0640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde70038e70 con 0x7fde700c9670 2024-03-29T11:49:04.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.945+0000 7fde803d0640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde700335d0 con 0x7fde70035e00 2024-03-29T11:49:04.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.945+0000 7fde803d0640 10 monclient(hunting): _renew_subs 2024-03-29T11:49:04.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.945+0000 7fde803d0640 10 monclient(hunting): authenticate will time out at 852.030723s 2024-03-29T11:49:04.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.951+0000 7fde7e16c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fde70006e30 0x7fde70032ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:50589/0 (socket says 172.21.15.98:50589) 2024-03-29T11:49:04.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.951+0000 7fde7e16c640 1 -- 172.21.15.98:0/2949185098 learned_addr learned my addr 172.21.15.98:0/2949185098 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:49:04.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.951+0000 7fde6effd640 10 client.?.objecter ms_handle_connect 0x7fde70006e30 2024-03-29T11:49:04.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.951+0000 7fde6effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:49:04.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.951+0000 7fde6effd640 1 -- 172.21.15.98:0/2949185098 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1362433057 0 0) 0x7fde70033980 con 0x7fde70006e30 2024-03-29T11:49:04.958 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.951+0000 7fde6effd640 10 monclient(hunting): _negotiate 2024-03-29T11:49:04.958 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.951+0000 7fde6effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:04.958 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.951+0000 7fde6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:04.958 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.951+0000 7fde6effd640 10 monclient(hunting): my global_id is 4257 2024-03-29T11:49:04.958 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.951+0000 7fde6effd640 1 -- 172.21.15.98:0/2949185098 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde5c004eb0 con 0x7fde70006e30 2024-03-29T11:49:04.963 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.955+0000 7fde6effd640 10 client.?.objecter ms_handle_connect 0x7fde70035e00 2024-03-29T11:49:04.963 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.955+0000 7fde6effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:49:04.963 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.955+0000 7fde6effd640 1 -- 172.21.15.98:0/2949185098 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1628423468 0 0) 0x7fde700335d0 con 0x7fde70035e00 2024-03-29T11:49:04.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.955+0000 7fde6effd640 10 monclient(hunting): _negotiate 2024-03-29T11:49:04.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.955+0000 7fde6effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:04.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.955+0000 7fde6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:04.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.955+0000 7fde6effd640 10 monclient(hunting): my global_id is 4256 2024-03-29T11:49:04.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.955+0000 7fde6effd640 1 -- 172.21.15.98:0/2949185098 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde70033980 con 0x7fde70035e00 2024-03-29T11:49:04.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.955+0000 7fde6effd640 10 client.?.objecter ms_handle_connect 0x7fde700c9670 2024-03-29T11:49:04.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.955+0000 7fde6effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:49:04.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.955+0000 7fde6effd640 1 -- 172.21.15.98:0/2949185098 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3920054650 0 0) 0x7fde5c004eb0 con 0x7fde70006e30 2024-03-29T11:49:04.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.955+0000 7fde6effd640 1 -- 172.21.15.98:0/2949185098 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde700335d0 con 0x7fde70006e30 2024-03-29T11:49:04.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.955+0000 7fde6effd640 1 -- 172.21.15.98:0/2949185098 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fde7804f400 con 0x7fde70006e30 2024-03-29T11:49:04.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.955+0000 7fde6effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:04.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.955+0000 7fde6effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:49:04.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.955+0000 7fde6effd640 10 monclient(hunting): dump: 2024-03-29T11:49:04.964 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:49:04.964 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:49:04.965 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:04.965 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:04.965 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:04.965 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:04.965 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:49:04.965 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:49:04.965 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:49:04.965 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:04.965 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.955+0000 7fde6effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:49:04.965 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.956+0000 7fde6effd640 1 -- 172.21.15.98:0/2949185098 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2852725366 0 0) 0x7fde700335d0 con 0x7fde70006e30 2024-03-29T11:49:04.965 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.956+0000 7fde6effd640 1 -- 172.21.15.98:0/2949185098 >> v1:172.21.15.98:6790/0 conn(0x7fde70035e00 legacy=0x7fde700c09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:04.965 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.956+0000 7fde6effd640 1 -- 172.21.15.98:0/2949185098 >> v1:172.21.15.121:6789/0 conn(0x7fde700c9670 legacy=0x7fde70160b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:04.965 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.956+0000 7fde6effd640 10 monclient: _finish_hunting 0 2024-03-29T11:49:04.965 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.956+0000 7fde6effd640 1 monclient: found mon.a 2024-03-29T11:49:04.965 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.956+0000 7fde6effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:49:04.965 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.956+0000 7fde6effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:49:04.965 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.956+0000 7fde6effd640 1 -- 172.21.15.98:0/2949185098 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde700c10c0 con 0x7fde70006e30 2024-03-29T11:49:04.966 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.956+0000 7fde6effd640 10 monclient: _finish_auth 0 2024-03-29T11:49:04.966 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.956+0000 7fde6effd640 10 monclient: _check_auth_tickets 2024-03-29T11:49:04.966 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.956+0000 7fde6effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:49:04.966 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.956+0000 7fde6effd640 1 -- 172.21.15.98:0/2949185098 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fde7804f730 con 0x7fde70006e30 2024-03-29T11:49:04.966 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.956+0000 7fde6effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:49:04.966 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.956+0000 7fde6effd640 1 -- 172.21.15.98:0/2949185098 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fde78068040 con 0x7fde70006e30 2024-03-29T11:49:04.966 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.956+0000 7fde6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:04.966 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.956+0000 7fde6effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:49:04.966 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.956+0000 7fde6effd640 10 monclient: dump: 2024-03-29T11:49:04.966 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:49:04.966 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:49:04.966 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:04.966 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:04.966 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:04.966 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:04.966 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:49:04.967 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:49:04.967 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:49:04.967 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:04.967 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.956+0000 7fde7c969640 4 set_mon_vals no callback set 2024-03-29T11:49:04.967 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.958+0000 7fde803d0640 5 monclient: authenticate success, global_id 4257 2024-03-29T11:49:04.967 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.958+0000 7fde803d0640 10 monclient: _renew_subs 2024-03-29T11:49:04.967 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.958+0000 7fde803d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:49:04.967 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.958+0000 7fde803d0640 1 -- 172.21.15.98:0/2949185098 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde700c1310 con 0x7fde70006e30 2024-03-29T11:49:04.968 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.958+0000 7fde803d0640 10 client.4257.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:49:04.968 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.958+0000 7fde803d0640 10 monclient: _renew_subs 2024-03-29T11:49:04.968 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.958+0000 7fde803d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:49:04.968 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.958+0000 7fde803d0640 1 -- 172.21.15.98:0/2949185098 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fde70032b10 con 0x7fde70006e30 2024-03-29T11:49:04.968 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.958+0000 7fde803d0640 1 librados: init done 2024-03-29T11:49:04.968 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.960+0000 7fde803d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:04.968 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.960+0000 7fde803d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:04.968 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.960+0000 7fde803d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:49:04.968 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.960+0000 7fde803d0640 1 -- 172.21.15.98:0/2949185098 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde70007b90 con 0x7fde70006e30 2024-03-29T11:49:04.968 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.962+0000 7fde6effd640 1 -- 172.21.15.98:0/2949185098 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7fde78075710 con 0x7fde70006e30 2024-03-29T11:49:04.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.963+0000 7fde7d96b640 10 monclient: get_auth_request con 0x7fde5c06dbb0 auth_method 0 2024-03-29T11:49:04.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.963+0000 7fde6effd640 10 client.4257.objecter ms_handle_connect 0x7fde5c06dbb0 2024-03-29T11:49:04.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.963+0000 7fde6effd640 1 -- 172.21.15.98:0/2949185098 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7fde780ff610 con 0x7fde70006e30 2024-03-29T11:49:04.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.963+0000 7fde6effd640 10 client.4257.objecter ms_dispatch 0x7fde78000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:49:04.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.963+0000 7fde6effd640 3 client.4257.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:49:04.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.963+0000 7fde6effd640 3 client.4257.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:49:04.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.963+0000 7fde6effd640 20 client.4257.objecter dump_active .. 0 homeless 2024-03-29T11:49:04.977 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.971+0000 7fde6effd640 1 -- 172.21.15.98:0/2949185098 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fde780caef0 con 0x7fde70006e30 2024-03-29T11:49:04.977 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.971+0000 7fde6effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:04.977 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:04.971+0000 7fde6effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:49:05.292 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:05.286+0000 7fde803d0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:05.292 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:05.286+0000 7fde803d0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:05.292 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:05.286+0000 7fde803d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:49:05.292 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:05.286+0000 7fde803d0640 1 -- 172.21.15.98:0/2949185098 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fde70033760 con 0x7fde70006e30 2024-03-29T11:49:05.293 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:05.288+0000 7fde6effd640 1 -- 172.21.15.98:0/2949185098 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7fde780cce20 con 0x7fde70006e30 2024-03-29T11:49:05.293 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:05.288+0000 7fde6effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:05.294 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:05.288+0000 7fde6effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:49:05.297 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:49:05.297 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:49:05.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:05.291+0000 7fde803d0640 10 librados: watch_flush enter 2024-03-29T11:49:05.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:05.291+0000 7fde803d0640 10 librados: watch_flush exit 2024-03-29T11:49:05.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:05.291+0000 7fde803d0640 20 client.4257.objecter shutdown clearing up homeless session... 2024-03-29T11:49:05.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:05.292+0000 7fde803d0640 10 client.4257.objecter successfully canceled tick 2024-03-29T11:49:05.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:05.292+0000 7fde803d0640 1 -- 172.21.15.98:0/2949185098 >> v1:172.21.15.98:6816/4065186974 conn(0x7fde5c06dbb0 legacy=0x7fde5c070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:05.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:05.292+0000 7fde803d0640 10 monclient: shutdown 2024-03-29T11:49:05.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:05.292+0000 7fde803d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:49:05.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:05.292+0000 7fde803d0640 1 -- 172.21.15.98:0/2949185098 >> v1:172.21.15.98:6789/0 conn(0x7fde70006e30 legacy=0x7fde70032ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:05.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:05.292+0000 7fde803d0640 1 -- 172.21.15.98:0/2949185098 shutdown_connections 2024-03-29T11:49:05.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:05.292+0000 7fde803d0640 1 -- 172.21.15.98:0/2949185098 >> 172.21.15.98:0/2949185098 conn(0x7fde70005c00 msgr2=0x7fde70005fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:49:05.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:05.292+0000 7fde803d0640 1 -- 172.21.15.98:0/2949185098 shutdown_connections 2024-03-29T11:49:05.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:05.292+0000 7fde803d0640 1 -- 172.21.15.98:0/2949185098 wait complete. 2024-03-29T11:49:05.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:05.292+0000 7fde803d0640 1 librados: shutdown 2024-03-29T11:49:05.315 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:49:12.121 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:49:12.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.267+0000 7fd85bfff640 10 monclient: get_monmap_and_config 2024-03-29T11:49:12.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.267+0000 7fd85bfff640 10 monclient: build_initial_monmap 2024-03-29T11:49:12.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.267+0000 7fd85bfff640 10 monclient: monmap: 2024-03-29T11:49:12.273 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:49:12.273 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:49:12.273 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:49:12.268136+0000 2024-03-29T11:49:12.273 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:49:12.268136+0000 2024-03-29T11:49:12.274 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:49:12.274 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:12.274 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:49:12.274 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:49:12.274 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:49:12.274 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:12.276 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.270+0000 7fd85bfff640 1 Processor -- start 2024-03-29T11:49:12.276 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.270+0000 7fd85bfff640 1 -- start start 2024-03-29T11:49:12.276 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.270+0000 7fd85bfff640 10 monclient: init 2024-03-29T11:49:12.277 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.271+0000 7fd85bfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:49:12.277 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.271+0000 7fd85bfff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:49:12.277 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.271+0000 7fd85bfff640 10 monclient(hunting): picked mon.noname-b con 0x7fd854006eb0 addr v1:172.21.15.98:6790/0 2024-03-29T11:49:12.278 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.271+0000 7fd85bfff640 10 monclient(hunting): picked mon.noname-c con 0x7fd854038f10 addr v1:172.21.15.121:6789/0 2024-03-29T11:49:12.278 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.271+0000 7fd85bfff640 10 monclient(hunting): picked mon.noname-a con 0x7fd854036690 addr v1:172.21.15.98:6789/0 2024-03-29T11:49:12.278 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.271+0000 7fd85bfff640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd854033c90 con 0x7fd854036690 2024-03-29T11:49:12.278 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.271+0000 7fd85bfff640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8540389f0 con 0x7fd854038f10 2024-03-29T11:49:12.278 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.271+0000 7fd85bfff640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd854030890 con 0x7fd854006eb0 2024-03-29T11:49:12.278 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.271+0000 7fd85bfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:49:12.278 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.271+0000 7fd85bfff640 10 monclient(hunting): authenticate will time out at 859.356586s 2024-03-29T11:49:12.281 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.275+0000 7fd85affd640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fd854006eb0 0x7fd854031760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:43741/0 (socket says 172.21.15.98:43741) 2024-03-29T11:49:12.281 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.275+0000 7fd85affd640 1 -- 172.21.15.98:0/3861712292 learned_addr learned my addr 172.21.15.98:0/3861712292 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:49:12.286 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.279+0000 7fd859ffb640 1 -- 172.21.15.98:0/3861712292 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1393769969 0 0) 0x7fd854030890 con 0x7fd854006eb0 2024-03-29T11:49:12.286 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.279+0000 7fd859ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:49:12.286 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.279+0000 7fd859ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:12.286 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.279+0000 7fd859ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:12.286 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.279+0000 7fd859ffb640 10 monclient(hunting): my global_id is 4259 2024-03-29T11:49:12.286 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.279+0000 7fd859ffb640 1 -- 172.21.15.98:0/3861712292 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd844003520 con 0x7fd854006eb0 2024-03-29T11:49:12.286 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.279+0000 7fd859ffb640 1 -- 172.21.15.98:0/3861712292 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2218849495 0 0) 0x7fd8540389f0 con 0x7fd854038f10 2024-03-29T11:49:12.286 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.279+0000 7fd859ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:49:12.287 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.279+0000 7fd859ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:12.287 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.279+0000 7fd859ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:12.287 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.279+0000 7fd859ffb640 10 monclient(hunting): my global_id is 4276 2024-03-29T11:49:12.287 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.279+0000 7fd859ffb640 1 -- 172.21.15.98:0/3861712292 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd854030890 con 0x7fd854038f10 2024-03-29T11:49:12.287 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.280+0000 7fd859ffb640 1 -- 172.21.15.98:0/3861712292 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1353396431 0 0) 0x7fd854030890 con 0x7fd854038f10 2024-03-29T11:49:12.287 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.280+0000 7fd859ffb640 1 -- 172.21.15.98:0/3861712292 >> v1:172.21.15.98:6790/0 conn(0x7fd854006eb0 legacy=0x7fd854031760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:12.287 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.280+0000 7fd859ffb640 1 -- 172.21.15.98:0/3861712292 >> v1:172.21.15.98:6789/0 conn(0x7fd854036690 legacy=0x7fd854036b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:12.287 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.280+0000 7fd859ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:49:12.287 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.280+0000 7fd859ffb640 1 monclient: found mon.noname-c 2024-03-29T11:49:12.287 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.280+0000 7fd859ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:49:12.287 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.280+0000 7fd859ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:49:12.287 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.280+0000 7fd859ffb640 1 -- 172.21.15.98:0/3861712292 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8540cbc40 con 0x7fd854038f10 2024-03-29T11:49:12.287 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.280+0000 7fd859ffb640 10 monclient: _finish_auth 0 2024-03-29T11:49:12.287 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.280+0000 7fd859ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:49:12.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.280+0000 7fd859ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:49:12.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.280+0000 7fd85bfff640 5 monclient: authenticate success, global_id 4276 2024-03-29T11:49:12.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.280+0000 7fd85bfff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:49:12.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.280+0000 7fd859ffb640 1 -- 172.21.15.98:0/3861712292 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fd850001670 con 0x7fd854038f10 2024-03-29T11:49:12.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.280+0000 7fd859ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:12.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.280+0000 7fd859ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:49:12.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.280+0000 7fd859ffb640 10 monclient: dump: 2024-03-29T11:49:12.288 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:49:12.288 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:49:12.288 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:12.288 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:12.288 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:12.288 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:12.288 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:49:12.289 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:49:12.289 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:49:12.289 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:12.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.280+0000 7fd85bfff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:49:12.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.281+0000 7fd859ffb640 1 -- 172.21.15.98:0/3861712292 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd850001850 con 0x7fd854038f10 2024-03-29T11:49:12.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.281+0000 7fd859ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:49:12.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.281+0000 7fd85bfff640 10 monclient: get_monmap_and_config success 2024-03-29T11:49:12.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.281+0000 7fd85bfff640 4 set_mon_vals no callback set 2024-03-29T11:49:12.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.282+0000 7fd85bfff640 10 monclient: shutdown 2024-03-29T11:49:12.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.282+0000 7fd85bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:49:12.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.282+0000 7fd85bfff640 1 -- 172.21.15.98:0/3861712292 >> v1:172.21.15.121:6789/0 conn(0x7fd854038f10 legacy=0x7fd854039330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:12.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.282+0000 7fd859ffb640 1 -- 172.21.15.98:0/3861712292 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fd850001670 con 0x7fd854038f10 2024-03-29T11:49:12.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.282+0000 7fd859ffb640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:49:12.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.283+0000 7fd85bfff640 1 -- 172.21.15.98:0/3861712292 shutdown_connections 2024-03-29T11:49:12.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.283+0000 7fd85bfff640 1 -- 172.21.15.98:0/3861712292 >> 172.21.15.98:0/3861712292 conn(0x7fd854005c20 msgr2=0x7fd854006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:49:12.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.284+0000 7fd85bfff640 1 -- 172.21.15.98:0/3861712292 shutdown_connections 2024-03-29T11:49:12.293 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.284+0000 7fd85bfff640 1 -- 172.21.15.98:0/3861712292 wait complete. 2024-03-29T11:49:12.293 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.285+0000 7fd85bfff640 10 monclient: build_initial_monmap 2024-03-29T11:49:12.293 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.285+0000 7fd85bfff640 10 monclient: monmap: 2024-03-29T11:49:12.293 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:49:12.294 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:49:12.294 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:49:12.294 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:49:12.294 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:49:12.294 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:12.294 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:49:12.294 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:49:12.294 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:49:12.294 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:12.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.285+0000 7fd85bfff640 1 librados: starting msgr at 2024-03-29T11:49:12.300 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.285+0000 7fd85bfff640 1 librados: starting objecter 2024-03-29T11:49:12.300 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.285+0000 7fd85bfff640 1 Processor -- start 2024-03-29T11:49:12.300 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.286+0000 7fd85bfff640 1 -- start start 2024-03-29T11:49:12.300 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.286+0000 7fd85bfff640 1 librados: setting wanted keys 2024-03-29T11:49:12.300 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.286+0000 7fd85bfff640 1 librados: calling monclient init 2024-03-29T11:49:12.300 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.286+0000 7fd85bfff640 10 monclient: init 2024-03-29T11:49:12.300 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.286+0000 7fd85bfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:49:12.300 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.286+0000 7fd85bfff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:49:12.300 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.286+0000 7fd85bfff640 10 monclient(hunting): picked mon.noname-a con 0x7fd854006eb0 addr v1:172.21.15.98:6789/0 2024-03-29T11:49:12.300 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.286+0000 7fd85bfff640 10 monclient(hunting): picked mon.noname-c con 0x7fd854036690 addr v1:172.21.15.98:6790/0 2024-03-29T11:49:12.300 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.290+0000 7fd85bfff640 10 monclient(hunting): picked mon.noname-b con 0x7fd854038f10 addr v1:172.21.15.121:6789/0 2024-03-29T11:49:12.300 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.290+0000 7fd85bfff640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8540cfbe0 con 0x7fd854006eb0 2024-03-29T11:49:12.300 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.290+0000 7fd85bfff640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8540d0070 con 0x7fd854038f10 2024-03-29T11:49:12.300 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.290+0000 7fd85bfff640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8540c3420 con 0x7fd854036690 2024-03-29T11:49:12.301 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.290+0000 7fd85bfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:49:12.301 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.290+0000 7fd85bfff640 10 monclient(hunting): authenticate will time out at 859.375339s 2024-03-29T11:49:12.301 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.290+0000 7fd85b7fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fd854038f10 0x7fd8540c2d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:40361/0 (socket says 172.21.15.98:40361) 2024-03-29T11:49:12.301 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.290+0000 7fd85b7fe640 1 -- 172.21.15.98:0/3337009992 learned_addr learned my addr 172.21.15.98:0/3337009992 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:49:12.301 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.291+0000 7fd843fff640 10 client.?.objecter ms_handle_connect 0x7fd854038f10 2024-03-29T11:49:12.301 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.291+0000 7fd843fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:49:12.301 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.291+0000 7fd843fff640 10 client.?.objecter ms_handle_connect 0x7fd854006eb0 2024-03-29T11:49:12.301 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.291+0000 7fd843fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:49:12.301 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.291+0000 7fd843fff640 1 -- 172.21.15.98:0/3337009992 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1887321027 0 0) 0x7fd8540d0070 con 0x7fd854038f10 2024-03-29T11:49:12.301 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.291+0000 7fd843fff640 10 monclient(hunting): _negotiate 2024-03-29T11:49:12.301 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.291+0000 7fd843fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:12.301 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.291+0000 7fd843fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:12.301 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.291+0000 7fd843fff640 10 monclient(hunting): my global_id is 4279 2024-03-29T11:49:12.301 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.291+0000 7fd843fff640 1 -- 172.21.15.98:0/3337009992 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd848004eb0 con 0x7fd854038f10 2024-03-29T11:49:12.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.291+0000 7fd843fff640 1 -- 172.21.15.98:0/3337009992 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2479168901 0 0) 0x7fd8540cfbe0 con 0x7fd854006eb0 2024-03-29T11:49:12.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.291+0000 7fd843fff640 10 monclient(hunting): _negotiate 2024-03-29T11:49:12.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.292+0000 7fd843fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:12.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.292+0000 7fd843fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:12.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.292+0000 7fd843fff640 10 monclient(hunting): my global_id is 4263 2024-03-29T11:49:12.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.292+0000 7fd843fff640 1 -- 172.21.15.98:0/3337009992 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8540d0070 con 0x7fd854006eb0 2024-03-29T11:49:12.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.292+0000 7fd843fff640 10 client.?.objecter ms_handle_connect 0x7fd854036690 2024-03-29T11:49:12.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.292+0000 7fd843fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:49:12.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.292+0000 7fd843fff640 1 -- 172.21.15.98:0/3337009992 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1334090122 0 0) 0x7fd8540c3420 con 0x7fd854036690 2024-03-29T11:49:12.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.292+0000 7fd843fff640 10 monclient(hunting): _negotiate 2024-03-29T11:49:12.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.292+0000 7fd843fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:12.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.292+0000 7fd843fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:12.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.292+0000 7fd843fff640 10 monclient(hunting): my global_id is 4262 2024-03-29T11:49:12.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.292+0000 7fd843fff640 1 -- 172.21.15.98:0/3337009992 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8540cfbe0 con 0x7fd854036690 2024-03-29T11:49:12.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.295+0000 7fd843fff640 1 -- 172.21.15.98:0/3337009992 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1632779421 0 0) 0x7fd848004eb0 con 0x7fd854038f10 2024-03-29T11:49:12.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.295+0000 7fd843fff640 1 -- 172.21.15.98:0/3337009992 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8540c3420 con 0x7fd854038f10 2024-03-29T11:49:12.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.295+0000 7fd843fff640 1 -- 172.21.15.98:0/3337009992 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3569435421 0 0) 0x7fd8540cfbe0 con 0x7fd854036690 2024-03-29T11:49:12.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.295+0000 7fd843fff640 1 -- 172.21.15.98:0/3337009992 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd848004eb0 con 0x7fd854036690 2024-03-29T11:49:12.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.295+0000 7fd843fff640 1 -- 172.21.15.98:0/3337009992 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fd85c04fdd0 con 0x7fd854038f10 2024-03-29T11:49:12.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.295+0000 7fd843fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:12.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.295+0000 7fd843fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:49:12.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.295+0000 7fd843fff640 10 monclient(hunting): dump: 2024-03-29T11:49:12.307 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:49:12.307 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:49:12.307 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:12.307 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:12.307 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:12.307 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:12.308 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:49:12.308 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:49:12.308 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:49:12.308 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:12.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.295+0000 7fd843fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:49:12.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.295+0000 7fd843fff640 1 -- 172.21.15.98:0/3337009992 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fd850001d40 con 0x7fd854036690 2024-03-29T11:49:12.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.295+0000 7fd843fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:12.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.295+0000 7fd843fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:49:12.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.296+0000 7fd843fff640 10 monclient(hunting): dump: 2024-03-29T11:49:12.308 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:49:12.308 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:49:12.308 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:12.308 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:12.308 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:12.309 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:12.309 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:49:12.309 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:49:12.309 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:49:12.309 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:12.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.296+0000 7fd843fff640 1 -- 172.21.15.98:0/3337009992 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1120583030 0 0) 0x7fd8540c3420 con 0x7fd854038f10 2024-03-29T11:49:12.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.296+0000 7fd843fff640 1 -- 172.21.15.98:0/3337009992 >> v1:172.21.15.98:6790/0 conn(0x7fd854036690 legacy=0x7fd8540c2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:12.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.296+0000 7fd843fff640 1 -- 172.21.15.98:0/3337009992 >> v1:172.21.15.98:6789/0 conn(0x7fd854006eb0 legacy=0x7fd8541643d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:12.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.296+0000 7fd843fff640 10 monclient: _finish_hunting 0 2024-03-29T11:49:12.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.296+0000 7fd843fff640 1 monclient: found mon.b 2024-03-29T11:49:12.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.296+0000 7fd843fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:49:12.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.297+0000 7fd843fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:49:12.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.297+0000 7fd843fff640 1 -- 172.21.15.98:0/3337009992 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd85400a8c0 con 0x7fd854038f10 2024-03-29T11:49:12.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.297+0000 7fd843fff640 10 monclient: _finish_auth 0 2024-03-29T11:49:12.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.297+0000 7fd843fff640 10 monclient: _check_auth_tickets 2024-03-29T11:49:12.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.297+0000 7fd843fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:49:12.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.297+0000 7fd843fff640 1 -- 172.21.15.98:0/3337009992 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd85c04f400 con 0x7fd854038f10 2024-03-29T11:49:12.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.297+0000 7fd843fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:49:12.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.297+0000 7fd843fff640 1 -- 172.21.15.98:0/3337009992 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fd85c075710 con 0x7fd854038f10 2024-03-29T11:49:12.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.297+0000 7fd843fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:12.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.297+0000 7fd843fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:49:12.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.298+0000 7fd8597fa640 4 set_mon_vals no callback set 2024-03-29T11:49:12.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.299+0000 7fd843fff640 10 monclient: dump: 2024-03-29T11:49:12.310 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:49:12.310 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:49:12.310 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:12.310 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:12.310 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:12.310 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:12.311 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:49:12.311 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:49:12.311 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:49:12.311 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:12.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.299+0000 7fd85bfff640 5 monclient: authenticate success, global_id 4279 2024-03-29T11:49:12.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.299+0000 7fd85bfff640 10 monclient: _renew_subs 2024-03-29T11:49:12.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.299+0000 7fd85bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:49:12.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.299+0000 7fd85bfff640 1 -- 172.21.15.98:0/3337009992 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8540cf730 con 0x7fd854038f10 2024-03-29T11:49:12.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.299+0000 7fd85bfff640 10 client.4279.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:49:12.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.299+0000 7fd85bfff640 10 monclient: _renew_subs 2024-03-29T11:49:12.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.299+0000 7fd85bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:49:12.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.299+0000 7fd85bfff640 1 -- 172.21.15.98:0/3337009992 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd85402fe10 con 0x7fd854038f10 2024-03-29T11:49:12.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.299+0000 7fd85bfff640 1 librados: init done 2024-03-29T11:49:12.313 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.302+0000 7fd843fff640 1 -- 172.21.15.98:0/3337009992 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7fd85c067bf0 con 0x7fd854038f10 2024-03-29T11:49:12.313 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.302+0000 7fd85affd640 10 monclient: get_auth_request con 0x7fd84806dbc0 auth_method 0 2024-03-29T11:49:12.313 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.303+0000 7fd843fff640 10 client.4279.objecter ms_handle_connect 0x7fd84806dbc0 2024-03-29T11:49:12.313 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.303+0000 7fd843fff640 1 -- 172.21.15.98:0/3337009992 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7fd85c110520 con 0x7fd854038f10 2024-03-29T11:49:12.313 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.303+0000 7fd843fff640 10 client.4279.objecter ms_dispatch 0x7fd85c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:49:12.313 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.303+0000 7fd843fff640 3 client.4279.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:49:12.313 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.303+0000 7fd843fff640 3 client.4279.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:49:12.313 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.303+0000 7fd843fff640 20 client.4279.objecter dump_active .. 0 homeless 2024-03-29T11:49:12.313 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.304+0000 7fd85bfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:12.313 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.304+0000 7fd85bfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:12.314 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.304+0000 7fd85bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:49:12.314 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.304+0000 7fd85bfff640 1 -- 172.21.15.98:0/3337009992 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd854007b70 con 0x7fd854038f10 2024-03-29T11:49:12.319 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.314+0000 7fd843fff640 1 -- 172.21.15.98:0/3337009992 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd85c0dbe00 con 0x7fd854038f10 2024-03-29T11:49:12.319 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.314+0000 7fd843fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:12.319 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.314+0000 7fd843fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:49:12.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.633+0000 7fd85bfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:12.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.633+0000 7fd85bfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:12.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.633+0000 7fd85bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:49:12.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.633+0000 7fd85bfff640 1 -- 172.21.15.98:0/3337009992 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd854004cd0 con 0x7fd854038f10 2024-03-29T11:49:12.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.634+0000 7fd843fff640 1 -- 172.21.15.98:0/3337009992 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7fd85c0ddd30 con 0x7fd854038f10 2024-03-29T11:49:12.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.634+0000 7fd843fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:12.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.634+0000 7fd843fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:49:12.643 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:49:12.643 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:49:12.645 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.640+0000 7fd85bfff640 10 librados: watch_flush enter 2024-03-29T11:49:12.646 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.640+0000 7fd85bfff640 10 librados: watch_flush exit 2024-03-29T11:49:12.646 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.640+0000 7fd85bfff640 20 client.4279.objecter shutdown clearing up homeless session... 2024-03-29T11:49:12.646 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.640+0000 7fd85bfff640 10 client.4279.objecter successfully canceled tick 2024-03-29T11:49:12.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.642+0000 7fd85bfff640 1 -- 172.21.15.98:0/3337009992 >> v1:172.21.15.98:6816/4065186974 conn(0x7fd84806dbc0 legacy=0x7fd848070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:12.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.642+0000 7fd85bfff640 10 monclient: shutdown 2024-03-29T11:49:12.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.642+0000 7fd85bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:49:12.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.642+0000 7fd85bfff640 1 -- 172.21.15.98:0/3337009992 >> v1:172.21.15.121:6789/0 conn(0x7fd854038f10 legacy=0x7fd8540c2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:12.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.643+0000 7fd85bfff640 1 -- 172.21.15.98:0/3337009992 shutdown_connections 2024-03-29T11:49:12.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.643+0000 7fd85bfff640 1 -- 172.21.15.98:0/3337009992 >> 172.21.15.98:0/3337009992 conn(0x7fd854005c20 msgr2=0x7fd8540050f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:49:12.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.643+0000 7fd85bfff640 1 -- 172.21.15.98:0/3337009992 shutdown_connections 2024-03-29T11:49:12.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.643+0000 7fd85bfff640 1 -- 172.21.15.98:0/3337009992 wait complete. 2024-03-29T11:49:12.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:12.643+0000 7fd85bfff640 1 librados: shutdown 2024-03-29T11:49:12.670 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:49:19.477 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:49:19.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.622+0000 7f4f5b59e640 10 monclient: get_monmap_and_config 2024-03-29T11:49:19.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.622+0000 7f4f5b59e640 10 monclient: build_initial_monmap 2024-03-29T11:49:19.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.622+0000 7f4f5b59e640 10 monclient: monmap: 2024-03-29T11:49:19.629 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:49:19.629 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:49:19.630 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:49:19.623506+0000 2024-03-29T11:49:19.630 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:49:19.623506+0000 2024-03-29T11:49:19.630 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:49:19.630 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:19.630 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:49:19.630 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:49:19.630 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:49:19.630 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:19.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.624+0000 7f4f5b59e640 1 Processor -- start 2024-03-29T11:49:19.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.624+0000 7f4f5b59e640 1 -- start start 2024-03-29T11:49:19.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.624+0000 7f4f5b59e640 10 monclient: init 2024-03-29T11:49:19.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.625+0000 7f4f5b59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:49:19.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.625+0000 7f4f5b59e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:49:19.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.625+0000 7f4f5b59e640 10 monclient(hunting): picked mon.noname-a con 0x7f4f54034850 addr v1:172.21.15.98:6789/0 2024-03-29T11:49:19.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.625+0000 7f4f5b59e640 10 monclient(hunting): picked mon.noname-c con 0x7f4f540c9550 addr v1:172.21.15.121:6789/0 2024-03-29T11:49:19.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.625+0000 7f4f5b59e640 10 monclient(hunting): picked mon.noname-b con 0x7f4f5402aac0 addr v1:172.21.15.98:6790/0 2024-03-29T11:49:19.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.625+0000 7f4f5b59e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f5402c440 con 0x7f4f54034850 2024-03-29T11:49:19.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.625+0000 7f4f5b59e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f5402aed0 con 0x7f4f540c9550 2024-03-29T11:49:19.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.625+0000 7f4f5b59e640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f5402b190 con 0x7f4f5402aac0 2024-03-29T11:49:19.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.625+0000 7f4f5b59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:49:19.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.625+0000 7f4f5b59e640 10 monclient(hunting): authenticate will time out at 866.710792s 2024-03-29T11:49:19.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.629+0000 7f4f5a59c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f4f54034850 0x7f4f540c8e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55735/0 (socket says 172.21.15.98:55735) 2024-03-29T11:49:19.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.629+0000 7f4f5a59c640 1 -- 172.21.15.98:0/872277399 learned_addr learned my addr 172.21.15.98:0/872277399 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:49:19.636 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.629+0000 7f4f5959a640 1 -- 172.21.15.98:0/872277399 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2844844496 0 0) 0x7f4f5402aed0 con 0x7f4f540c9550 2024-03-29T11:49:19.636 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.630+0000 7f4f5959a640 10 monclient(hunting): _negotiate 2024-03-29T11:49:19.636 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.630+0000 7f4f5959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:19.636 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.630+0000 7f4f5959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:19.636 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.630+0000 7f4f5959a640 10 monclient(hunting): my global_id is 4282 2024-03-29T11:49:19.637 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.630+0000 7f4f5959a640 1 -- 172.21.15.98:0/872277399 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f44003520 con 0x7f4f540c9550 2024-03-29T11:49:19.637 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.630+0000 7f4f5959a640 1 -- 172.21.15.98:0/872277399 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1585426105 0 0) 0x7f4f5402c440 con 0x7f4f54034850 2024-03-29T11:49:19.637 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.630+0000 7f4f5959a640 10 monclient(hunting): _negotiate 2024-03-29T11:49:19.637 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.630+0000 7f4f5959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:19.637 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.630+0000 7f4f5959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:19.637 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.630+0000 7f4f5959a640 10 monclient(hunting): my global_id is 4266 2024-03-29T11:49:19.637 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.630+0000 7f4f5959a640 1 -- 172.21.15.98:0/872277399 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f5402aed0 con 0x7f4f54034850 2024-03-29T11:49:19.643 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.631+0000 7f4f5959a640 1 -- 172.21.15.98:0/872277399 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 222686110 0 0) 0x7f4f44003520 con 0x7f4f540c9550 2024-03-29T11:49:19.643 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.631+0000 7f4f5959a640 1 -- 172.21.15.98:0/872277399 >> v1:172.21.15.98:6790/0 conn(0x7f4f5402aac0 legacy=0x7f4f540cc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:19.643 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.631+0000 7f4f5959a640 1 -- 172.21.15.98:0/872277399 >> v1:172.21.15.98:6789/0 conn(0x7f4f54034850 legacy=0x7f4f540c8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:19.643 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.631+0000 7f4f5959a640 10 monclient: _finish_hunting 0 2024-03-29T11:49:19.643 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.631+0000 7f4f5959a640 1 monclient: found mon.noname-c 2024-03-29T11:49:19.643 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.631+0000 7f4f5959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:49:19.643 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.631+0000 7f4f5959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:49:19.643 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.631+0000 7f4f5959a640 1 -- 172.21.15.98:0/872277399 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f54033060 con 0x7f4f540c9550 2024-03-29T11:49:19.644 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.631+0000 7f4f5959a640 10 monclient: _finish_auth 0 2024-03-29T11:49:19.644 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.631+0000 7f4f5959a640 10 monclient: _check_auth_tickets 2024-03-29T11:49:19.644 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.631+0000 7f4f5959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:49:19.644 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.631+0000 7f4f5b59e640 5 monclient: authenticate success, global_id 4282 2024-03-29T11:49:19.644 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.632+0000 7f4f5b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:49:19.644 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.632+0000 7f4f5959a640 1 -- 172.21.15.98:0/872277399 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f4f4c001690 con 0x7f4f540c9550 2024-03-29T11:49:19.644 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.632+0000 7f4f5959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:19.644 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.632+0000 7f4f5959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:49:19.644 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.632+0000 7f4f5959a640 10 monclient: dump: 2024-03-29T11:49:19.644 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:49:19.644 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:49:19.644 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:19.644 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:19.645 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:19.645 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:19.645 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:49:19.645 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:49:19.645 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:49:19.645 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:19.645 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.632+0000 7f4f5b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:49:19.645 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.632+0000 7f4f5959a640 1 -- 172.21.15.98:0/872277399 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f4c0017f0 con 0x7f4f540c9550 2024-03-29T11:49:19.645 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.632+0000 7f4f5959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:49:19.645 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.632+0000 7f4f5b59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:49:19.645 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.632+0000 7f4f5b59e640 4 set_mon_vals no callback set 2024-03-29T11:49:19.645 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.633+0000 7f4f5b59e640 10 monclient: shutdown 2024-03-29T11:49:19.645 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.633+0000 7f4f5b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:49:19.646 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.633+0000 7f4f5b59e640 1 -- 172.21.15.98:0/872277399 >> v1:172.21.15.121:6789/0 conn(0x7f4f540c9550 legacy=0x7f4f540cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:19.646 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.633+0000 7f4f5b59e640 1 -- 172.21.15.98:0/872277399 shutdown_connections 2024-03-29T11:49:19.646 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.633+0000 7f4f5b59e640 1 -- 172.21.15.98:0/872277399 >> 172.21.15.98:0/872277399 conn(0x7f4f540058f0 msgr2=0x7f4f54005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:49:19.646 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.633+0000 7f4f5b59e640 1 -- 172.21.15.98:0/872277399 shutdown_connections 2024-03-29T11:49:19.646 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.633+0000 7f4f5b59e640 1 -- 172.21.15.98:0/872277399 wait complete. 2024-03-29T11:49:19.646 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.634+0000 7f4f5b59e640 10 monclient: build_initial_monmap 2024-03-29T11:49:19.646 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.634+0000 7f4f5b59e640 10 monclient: monmap: 2024-03-29T11:49:19.646 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:49:19.646 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:49:19.646 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:49:19.646 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:49:19.647 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:49:19.647 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:19.647 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:49:19.647 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:49:19.647 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:49:19.647 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:19.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.635+0000 7f4f5b59e640 1 librados: starting msgr at 2024-03-29T11:49:19.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.635+0000 7f4f5b59e640 1 librados: starting objecter 2024-03-29T11:49:19.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.635+0000 7f4f5b59e640 1 Processor -- start 2024-03-29T11:49:19.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.635+0000 7f4f5b59e640 1 -- start start 2024-03-29T11:49:19.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.635+0000 7f4f5b59e640 1 librados: setting wanted keys 2024-03-29T11:49:19.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.635+0000 7f4f5b59e640 1 librados: calling monclient init 2024-03-29T11:49:19.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.635+0000 7f4f5b59e640 10 monclient: init 2024-03-29T11:49:19.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.636+0000 7f4f5b59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:49:19.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.636+0000 7f4f5b59e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:49:19.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.636+0000 7f4f5b59e640 10 monclient(hunting): picked mon.noname-a con 0x7f4f5402aac0 addr v1:172.21.15.98:6789/0 2024-03-29T11:49:19.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.636+0000 7f4f5b59e640 10 monclient(hunting): picked mon.noname-b con 0x7f4f54034850 addr v1:172.21.15.121:6789/0 2024-03-29T11:49:19.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.636+0000 7f4f5b59e640 10 monclient(hunting): picked mon.noname-c con 0x7f4f540c9550 addr v1:172.21.15.98:6790/0 2024-03-29T11:49:19.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.636+0000 7f4f5b59e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f54160ce0 con 0x7f4f5402aac0 2024-03-29T11:49:19.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.636+0000 7f4f5b59e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f540c0d30 con 0x7f4f54034850 2024-03-29T11:49:19.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.636+0000 7f4f5b59e640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f540bfe20 con 0x7f4f540c9550 2024-03-29T11:49:19.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.636+0000 7f4f5b59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:49:19.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.636+0000 7f4f5b59e640 10 monclient(hunting): authenticate will time out at 866.721452s 2024-03-29T11:49:19.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.641+0000 7f4f59d9b640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f4f54034850 0x7f4f54161430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:43061/0 (socket says 172.21.15.98:43061) 2024-03-29T11:49:19.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.641+0000 7f4f59d9b640 1 -- 172.21.15.98:0/1248136015 learned_addr learned my addr 172.21.15.98:0/1248136015 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:49:19.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.642+0000 7f4f4affd640 10 client.?.objecter ms_handle_connect 0x7f4f540c9550 2024-03-29T11:49:19.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.642+0000 7f4f4affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:49:19.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.642+0000 7f4f4affd640 1 -- 172.21.15.98:0/1248136015 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1275878694 0 0) 0x7f4f540bfe20 con 0x7f4f540c9550 2024-03-29T11:49:19.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.642+0000 7f4f4affd640 10 monclient(hunting): _negotiate 2024-03-29T11:49:19.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.642+0000 7f4f4affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:19.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.642+0000 7f4f4affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:19.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.642+0000 7f4f4affd640 10 monclient(hunting): my global_id is 4265 2024-03-29T11:49:19.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.642+0000 7f4f4affd640 1 -- 172.21.15.98:0/1248136015 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f3c004eb0 con 0x7f4f540c9550 2024-03-29T11:49:19.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.643+0000 7f4f4affd640 1 -- 172.21.15.98:0/1248136015 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4172969334 0 0) 0x7f4f3c004eb0 con 0x7f4f540c9550 2024-03-29T11:49:19.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.643+0000 7f4f4affd640 1 -- 172.21.15.98:0/1248136015 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f3c005300 con 0x7f4f540c9550 2024-03-29T11:49:19.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.643+0000 7f4f4affd640 10 client.?.objecter ms_handle_connect 0x7f4f54034850 2024-03-29T11:49:19.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.643+0000 7f4f4affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:49:19.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.643+0000 7f4f4affd640 1 -- 172.21.15.98:0/1248136015 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f4f5c04fdd0 con 0x7f4f540c9550 2024-03-29T11:49:19.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.643+0000 7f4f4affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:19.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.643+0000 7f4f4affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:49:19.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.643+0000 7f4f4affd640 10 monclient(hunting): dump: 2024-03-29T11:49:19.651 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:49:19.651 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:49:19.651 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:19.651 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:19.651 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:19.651 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:19.651 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:49:19.651 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:49:19.651 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:49:19.651 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:19.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.643+0000 7f4f4affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:49:19.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.645+0000 7f4f4affd640 1 -- 172.21.15.98:0/1248136015 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2125182564 0 0) 0x7f4f3c005300 con 0x7f4f540c9550 2024-03-29T11:49:19.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.645+0000 7f4f4affd640 1 -- 172.21.15.98:0/1248136015 >> v1:172.21.15.121:6789/0 conn(0x7f4f54034850 legacy=0x7f4f54161430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:19.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.645+0000 7f4f4affd640 1 -- 172.21.15.98:0/1248136015 >> v1:172.21.15.98:6789/0 conn(0x7f4f5402aac0 legacy=0x7f4f540bf710 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:49:19.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.645+0000 7f4f4affd640 10 monclient: _finish_hunting 0 2024-03-29T11:49:19.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.645+0000 7f4f4affd640 1 monclient: found mon.c 2024-03-29T11:49:19.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.645+0000 7f4f4affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:49:19.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.645+0000 7f4f4affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:49:19.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.645+0000 7f4f4affd640 1 -- 172.21.15.98:0/1248136015 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f54160f30 con 0x7f4f540c9550 2024-03-29T11:49:19.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.645+0000 7f4f4affd640 10 monclient: _finish_auth 0 2024-03-29T11:49:19.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.645+0000 7f4f4affd640 10 monclient: _check_auth_tickets 2024-03-29T11:49:19.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.645+0000 7f4f4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:49:19.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.645+0000 7f4f5b59e640 5 monclient: authenticate success, global_id 4265 2024-03-29T11:49:19.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.645+0000 7f4f5b59e640 10 monclient: _renew_subs 2024-03-29T11:49:19.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.645+0000 7f4f5b59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:49:19.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.645+0000 7f4f5b59e640 1 -- 172.21.15.98:0/1248136015 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f54161b40 con 0x7f4f540c9550 2024-03-29T11:49:19.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.645+0000 7f4f5b59e640 10 client.4265.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:49:19.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.645+0000 7f4f5b59e640 10 monclient: _renew_subs 2024-03-29T11:49:19.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.645+0000 7f4f5b59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:49:19.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.645+0000 7f4f5b59e640 1 -- 172.21.15.98:0/1248136015 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f540bedf0 con 0x7f4f540c9550 2024-03-29T11:49:19.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.645+0000 7f4f5b59e640 1 librados: init done 2024-03-29T11:49:19.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.646+0000 7f4f5b59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:19.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.646+0000 7f4f5b59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:19.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.647+0000 7f4f5b59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:49:19.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.647+0000 7f4f5b59e640 1 -- 172.21.15.98:0/1248136015 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f540bf0a0 con 0x7f4f540c9550 2024-03-29T11:49:19.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.647+0000 7f4f4affd640 1 -- 172.21.15.98:0/1248136015 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f5c04f400 con 0x7f4f540c9550 2024-03-29T11:49:19.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.647+0000 7f4f4affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:49:19.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.647+0000 7f4f4affd640 1 -- 172.21.15.98:0/1248136015 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f4f5c075f10 con 0x7f4f540c9550 2024-03-29T11:49:19.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.647+0000 7f4f4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:19.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.647+0000 7f4f4affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:49:19.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.647+0000 7f4f4affd640 10 monclient: dump: 2024-03-29T11:49:19.655 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:49:19.655 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:49:19.655 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:19.655 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:19.655 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:19.655 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:19.655 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:49:19.656 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:49:19.656 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:49:19.656 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:19.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.647+0000 7f4f58d99640 4 set_mon_vals no callback set 2024-03-29T11:49:19.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.648+0000 7f4f4affd640 1 -- 172.21.15.98:0/1248136015 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f4f5c067bf0 con 0x7f4f540c9550 2024-03-29T11:49:19.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.648+0000 7f4f4affd640 1 -- 172.21.15.98:0/1248136015 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f4f5c0fe270 con 0x7f4f540c9550 2024-03-29T11:49:19.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.648+0000 7f4f4affd640 10 client.4265.objecter ms_dispatch 0x7f4f5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:49:19.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.648+0000 7f4f4affd640 3 client.4265.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:49:19.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.648+0000 7f4f4affd640 3 client.4265.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:49:19.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.648+0000 7f4f4affd640 20 client.4265.objecter dump_active .. 0 homeless 2024-03-29T11:49:19.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.649+0000 7f4f5a59c640 10 monclient: get_auth_request con 0x7f4f3c06de50 auth_method 0 2024-03-29T11:49:19.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.652+0000 7f4f4affd640 10 client.4265.objecter ms_handle_connect 0x7f4f3c06de50 2024-03-29T11:49:19.666 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.660+0000 7f4f4affd640 1 -- 172.21.15.98:0/1248136015 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4f5c0c9c10 con 0x7f4f540c9550 2024-03-29T11:49:19.666 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.660+0000 7f4f4affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:19.666 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.660+0000 7f4f4affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:49:19.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.980+0000 7f4f5b59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:19.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.980+0000 7f4f5b59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:19.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.980+0000 7f4f5b59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:49:19.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.980+0000 7f4f5b59e640 1 -- 172.21.15.98:0/1248136015 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f54004cd0 con 0x7f4f540c9550 2024-03-29T11:49:19.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.983+0000 7f4f4affd640 1 -- 172.21.15.98:0/1248136015 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f4f5c0c9c10 con 0x7f4f540c9550 2024-03-29T11:49:19.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.983+0000 7f4f4affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:19.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.983+0000 7f4f4affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:49:19.990 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:49:19.990 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:49:19.994 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.988+0000 7f4f5b59e640 10 librados: watch_flush enter 2024-03-29T11:49:19.994 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.988+0000 7f4f5b59e640 10 librados: watch_flush exit 2024-03-29T11:49:19.994 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.988+0000 7f4f5b59e640 20 client.4265.objecter shutdown clearing up homeless session... 2024-03-29T11:49:19.994 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.988+0000 7f4f5b59e640 10 client.4265.objecter successfully canceled tick 2024-03-29T11:49:19.994 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.988+0000 7f4f5b59e640 1 -- 172.21.15.98:0/1248136015 >> v1:172.21.15.98:6816/4065186974 conn(0x7f4f3c06de50 legacy=0x7f4f3c070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:19.994 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.988+0000 7f4f5b59e640 10 monclient: shutdown 2024-03-29T11:49:19.994 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.988+0000 7f4f5b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:49:19.994 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.988+0000 7f4f5b59e640 1 -- 172.21.15.98:0/1248136015 >> v1:172.21.15.98:6790/0 conn(0x7f4f540c9550 legacy=0x7f4f5416b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:19.994 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.988+0000 7f4f5b59e640 1 -- 172.21.15.98:0/1248136015 shutdown_connections 2024-03-29T11:49:19.995 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.988+0000 7f4f5b59e640 1 -- 172.21.15.98:0/1248136015 >> 172.21.15.98:0/1248136015 conn(0x7f4f540058f0 msgr2=0x7f4f540c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:49:19.995 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.988+0000 7f4f5b59e640 1 -- 172.21.15.98:0/1248136015 shutdown_connections 2024-03-29T11:49:19.995 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.989+0000 7f4f5b59e640 1 -- 172.21.15.98:0/1248136015 wait complete. 2024-03-29T11:49:19.999 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:19.993+0000 7f4f5b59e640 1 librados: shutdown 2024-03-29T11:49:20.017 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:49:26.824 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:49:26.970 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.964+0000 7fb3970f6640 10 monclient: get_monmap_and_config 2024-03-29T11:49:26.970 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.964+0000 7fb3970f6640 10 monclient: build_initial_monmap 2024-03-29T11:49:26.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.964+0000 7fb3970f6640 10 monclient: monmap: 2024-03-29T11:49:26.971 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:49:26.971 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:49:26.971 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:49:26.965166+0000 2024-03-29T11:49:26.971 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:49:26.965166+0000 2024-03-29T11:49:26.971 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:49:26.971 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:26.971 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:49:26.971 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:49:26.971 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:49:26.972 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:26.973 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.965+0000 7fb3970f6640 1 Processor -- start 2024-03-29T11:49:26.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.965+0000 7fb3970f6640 1 -- start start 2024-03-29T11:49:26.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.965+0000 7fb3970f6640 10 monclient: init 2024-03-29T11:49:26.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.966+0000 7fb3970f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:49:26.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.966+0000 7fb3970f6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:49:26.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.966+0000 7fb3970f6640 10 monclient(hunting): picked mon.noname-a con 0x7fb3900b0770 addr v1:172.21.15.98:6789/0 2024-03-29T11:49:26.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.967+0000 7fb3970f6640 10 monclient(hunting): picked mon.noname-b con 0x7fb39014a700 addr v1:172.21.15.98:6790/0 2024-03-29T11:49:26.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.967+0000 7fb3970f6640 10 monclient(hunting): picked mon.noname-c con 0x7fb3900ac5a0 addr v1:172.21.15.121:6789/0 2024-03-29T11:49:26.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.967+0000 7fb3970f6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3900aba70 con 0x7fb3900b0770 2024-03-29T11:49:26.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.967+0000 7fb3970f6640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3900aecb0 con 0x7fb3900ac5a0 2024-03-29T11:49:26.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.967+0000 7fb3970f6640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3900af150 con 0x7fb39014a700 2024-03-29T11:49:26.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.967+0000 7fb3970f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:49:26.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.967+0000 7fb3970f6640 10 monclient(hunting): authenticate will time out at 874.052530s 2024-03-29T11:49:26.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.971+0000 7fb395693640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fb3900ac5a0 0x7fb390151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:58943/0 (socket says 172.21.15.98:58943) 2024-03-29T11:49:26.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.971+0000 7fb395693640 1 -- 172.21.15.98:0/1330146297 learned_addr learned my addr 172.21.15.98:0/1330146297 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:49:26.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.972+0000 7fb38f7fe640 1 -- 172.21.15.98:0/1330146297 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2314496104 0 0) 0x7fb3900aecb0 con 0x7fb3900ac5a0 2024-03-29T11:49:26.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.972+0000 7fb38f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:49:26.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.972+0000 7fb38f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:26.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.972+0000 7fb38f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:26.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.972+0000 7fb38f7fe640 10 monclient(hunting): my global_id is 4288 2024-03-29T11:49:26.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.972+0000 7fb38f7fe640 1 -- 172.21.15.98:0/1330146297 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb380003520 con 0x7fb3900ac5a0 2024-03-29T11:49:26.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.972+0000 7fb38f7fe640 1 -- 172.21.15.98:0/1330146297 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4114932220 0 0) 0x7fb380003520 con 0x7fb3900ac5a0 2024-03-29T11:49:26.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.972+0000 7fb38f7fe640 1 -- 172.21.15.98:0/1330146297 >> v1:172.21.15.98:6790/0 conn(0x7fb39014a700 legacy=0x7fb3900abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:26.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.972+0000 7fb38f7fe640 1 -- 172.21.15.98:0/1330146297 >> v1:172.21.15.98:6789/0 conn(0x7fb3900b0770 legacy=0x7fb390149ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:49:26.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.973+0000 7fb38f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:49:26.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.973+0000 7fb38f7fe640 1 monclient: found mon.noname-c 2024-03-29T11:49:26.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.973+0000 7fb38f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:49:26.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.973+0000 7fb38f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:49:26.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.973+0000 7fb38f7fe640 1 -- 172.21.15.98:0/1330146297 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3900ac9d0 con 0x7fb3900ac5a0 2024-03-29T11:49:26.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.973+0000 7fb38f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:49:26.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.973+0000 7fb38f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:49:26.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.973+0000 7fb38f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:49:26.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.973+0000 7fb38f7fe640 1 -- 172.21.15.98:0/1330146297 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fb388001a30 con 0x7fb3900ac5a0 2024-03-29T11:49:26.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.973+0000 7fb38f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:26.981 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.973+0000 7fb38f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:49:26.981 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.973+0000 7fb38f7fe640 10 monclient: dump: 2024-03-29T11:49:26.981 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:49:26.981 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:49:26.981 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:26.981 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:26.981 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:26.981 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:26.981 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:49:26.981 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:49:26.981 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:49:26.981 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:26.981 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.973+0000 7fb3970f6640 5 monclient: authenticate success, global_id 4288 2024-03-29T11:49:26.981 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.973+0000 7fb3970f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:49:26.982 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.974+0000 7fb38f7fe640 1 -- 172.21.15.98:0/1330146297 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb388001cc0 con 0x7fb3900ac5a0 2024-03-29T11:49:26.982 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.974+0000 7fb38f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:49:26.982 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.974+0000 7fb38f7fe640 1 -- 172.21.15.98:0/1330146297 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fb388002090 con 0x7fb3900ac5a0 2024-03-29T11:49:26.982 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.974+0000 7fb38f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:26.982 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.974+0000 7fb38f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:49:26.982 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.974+0000 7fb38f7fe640 10 monclient: dump: 2024-03-29T11:49:26.983 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:49:26.983 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:49:26.983 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:26.983 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:26.983 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:26.983 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:26.983 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:49:26.983 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:49:26.983 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:49:26.983 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:26.983 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.974+0000 7fb3970f6640 10 monclient: get_monmap_and_config success 2024-03-29T11:49:26.983 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.974+0000 7fb3970f6640 4 set_mon_vals no callback set 2024-03-29T11:49:26.983 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.975+0000 7fb3970f6640 10 monclient: shutdown 2024-03-29T11:49:26.984 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.975+0000 7fb3970f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:49:26.984 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.975+0000 7fb3970f6640 1 -- 172.21.15.98:0/1330146297 >> v1:172.21.15.121:6789/0 conn(0x7fb3900ac5a0 legacy=0x7fb390151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:26.984 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.977+0000 7fb3970f6640 1 -- 172.21.15.98:0/1330146297 shutdown_connections 2024-03-29T11:49:26.984 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.977+0000 7fb3970f6640 1 -- 172.21.15.98:0/1330146297 >> 172.21.15.98:0/1330146297 conn(0x7fb39006b660 msgr2=0x7fb39006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:49:26.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.980+0000 7fb3970f6640 1 -- 172.21.15.98:0/1330146297 shutdown_connections 2024-03-29T11:49:26.988 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.981+0000 7fb3970f6640 1 -- 172.21.15.98:0/1330146297 wait complete. 2024-03-29T11:49:26.988 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.982+0000 7fb3970f6640 10 monclient: build_initial_monmap 2024-03-29T11:49:26.988 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.982+0000 7fb3970f6640 10 monclient: monmap: 2024-03-29T11:49:26.988 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:49:26.988 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:49:26.988 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:49:26.988 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:49:26.988 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:49:26.989 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:26.989 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:49:26.989 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:49:26.989 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:49:26.989 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:26.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.983+0000 7fb3970f6640 1 librados: starting msgr at 2024-03-29T11:49:26.990 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.984+0000 7fb3970f6640 1 librados: starting objecter 2024-03-29T11:49:26.990 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.984+0000 7fb3970f6640 1 Processor -- start 2024-03-29T11:49:26.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.986+0000 7fb3970f6640 1 -- start start 2024-03-29T11:49:26.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.986+0000 7fb3970f6640 1 librados: setting wanted keys 2024-03-29T11:49:26.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.986+0000 7fb3970f6640 1 librados: calling monclient init 2024-03-29T11:49:26.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.986+0000 7fb3970f6640 10 monclient: init 2024-03-29T11:49:26.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.987+0000 7fb3970f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:49:26.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.987+0000 7fb3970f6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:49:26.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.987+0000 7fb3970f6640 10 monclient(hunting): picked mon.noname-c con 0x7fb3900ac5a0 addr v1:172.21.15.98:6790/0 2024-03-29T11:49:26.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.991+0000 7fb3970f6640 10 monclient(hunting): picked mon.noname-b con 0x7fb3900b0770 addr v1:172.21.15.121:6789/0 2024-03-29T11:49:26.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.991+0000 7fb3970f6640 10 monclient(hunting): picked mon.noname-a con 0x7fb39014a700 addr v1:172.21.15.98:6789/0 2024-03-29T11:49:26.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.991+0000 7fb3970f6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb390127490 con 0x7fb39014a700 2024-03-29T11:49:26.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.991+0000 7fb3970f6640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3901281a0 con 0x7fb3900b0770 2024-03-29T11:49:26.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.991+0000 7fb3970f6640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb390126090 con 0x7fb3900ac5a0 2024-03-29T11:49:26.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.991+0000 7fb3970f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:49:26.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.991+0000 7fb3970f6640 10 monclient(hunting): authenticate will time out at 874.076394s 2024-03-29T11:49:26.999 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.993+0000 7fb394e92640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fb3900ac5a0 0x7fb390126ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:44429/0 (socket says 172.21.15.98:44429) 2024-03-29T11:49:26.999 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.993+0000 7fb394e92640 1 -- 172.21.15.98:0/3865480693 learned_addr learned my addr 172.21.15.98:0/3865480693 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:49:27.006 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.997+0000 7fb38d7fa640 10 client.?.objecter ms_handle_connect 0x7fb3900ac5a0 2024-03-29T11:49:27.006 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.997+0000 7fb38d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:49:27.006 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.997+0000 7fb38d7fa640 10 client.?.objecter ms_handle_connect 0x7fb39014a700 2024-03-29T11:49:27.006 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.997+0000 7fb38d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:49:27.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.997+0000 7fb38d7fa640 10 client.?.objecter ms_handle_connect 0x7fb3900b0770 2024-03-29T11:49:27.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.997+0000 7fb38d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:49:27.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.997+0000 7fb38d7fa640 1 -- 172.21.15.98:0/3865480693 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 409089308 0 0) 0x7fb390126090 con 0x7fb3900ac5a0 2024-03-29T11:49:27.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.997+0000 7fb38d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:49:27.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.997+0000 7fb38d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:27.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.997+0000 7fb38d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:27.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.997+0000 7fb38d7fa640 10 monclient(hunting): my global_id is 4271 2024-03-29T11:49:27.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.997+0000 7fb38d7fa640 1 -- 172.21.15.98:0/3865480693 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb374004eb0 con 0x7fb3900ac5a0 2024-03-29T11:49:27.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.998+0000 7fb38d7fa640 1 -- 172.21.15.98:0/3865480693 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2058007342 0 0) 0x7fb390127490 con 0x7fb39014a700 2024-03-29T11:49:27.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.998+0000 7fb38d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:49:27.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.998+0000 7fb38d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:27.008 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.998+0000 7fb38d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:27.008 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.998+0000 7fb38d7fa640 10 monclient(hunting): my global_id is 4269 2024-03-29T11:49:27.008 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.998+0000 7fb38d7fa640 1 -- 172.21.15.98:0/3865480693 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb390126090 con 0x7fb39014a700 2024-03-29T11:49:27.008 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.998+0000 7fb38d7fa640 1 -- 172.21.15.98:0/3865480693 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3633175904 0 0) 0x7fb390126090 con 0x7fb39014a700 2024-03-29T11:49:27.008 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.998+0000 7fb38d7fa640 1 -- 172.21.15.98:0/3865480693 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb390127490 con 0x7fb39014a700 2024-03-29T11:49:27.008 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.998+0000 7fb38d7fa640 1 -- 172.21.15.98:0/3865480693 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fb388001850 con 0x7fb39014a700 2024-03-29T11:49:27.008 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.998+0000 7fb38d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:27.008 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.998+0000 7fb38d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:49:27.008 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.998+0000 7fb38d7fa640 10 monclient(hunting): dump: 2024-03-29T11:49:27.008 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:49:27.008 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:49:27.008 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:27.009 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:27.009 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:27.009 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:27.009 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:49:27.009 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:49:27.009 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:49:27.009 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:27.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.998+0000 7fb38d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:49:27.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.998+0000 7fb38d7fa640 1 -- 172.21.15.98:0/3865480693 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3338307258 0 0) 0x7fb390127490 con 0x7fb39014a700 2024-03-29T11:49:27.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.998+0000 7fb38d7fa640 1 -- 172.21.15.98:0/3865480693 >> v1:172.21.15.98:6790/0 conn(0x7fb3900ac5a0 legacy=0x7fb390126ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:27.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.998+0000 7fb38d7fa640 1 -- 172.21.15.98:0/3865480693 >> v1:172.21.15.121:6789/0 conn(0x7fb3900b0770 legacy=0x7fb3901dee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:27.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.998+0000 7fb38d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:49:27.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.998+0000 7fb38d7fa640 1 monclient: found mon.a 2024-03-29T11:49:27.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.998+0000 7fb38d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:49:27.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.998+0000 7fb38d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:49:27.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.998+0000 7fb38d7fa640 1 -- 172.21.15.98:0/3865480693 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb390126220 con 0x7fb39014a700 2024-03-29T11:49:27.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.998+0000 7fb38d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:49:27.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.998+0000 7fb38d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:49:27.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.998+0000 7fb38d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:49:27.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.998+0000 7fb3970f6640 5 monclient: authenticate success, global_id 4269 2024-03-29T11:49:27.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.998+0000 7fb3970f6640 10 monclient: _renew_subs 2024-03-29T11:49:27.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.998+0000 7fb3970f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:49:27.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.998+0000 7fb3970f6640 1 -- 172.21.15.98:0/3865480693 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3901df520 con 0x7fb39014a700 2024-03-29T11:49:27.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.998+0000 7fb3970f6640 10 client.4269.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:49:27.011 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.998+0000 7fb3970f6640 10 monclient: _renew_subs 2024-03-29T11:49:27.011 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.998+0000 7fb3970f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:49:27.011 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.998+0000 7fb3970f6640 1 -- 172.21.15.98:0/3865480693 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3901dfa00 con 0x7fb39014a700 2024-03-29T11:49:27.011 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:26.999+0000 7fb3970f6640 1 librados: init done 2024-03-29T11:49:27.011 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.000+0000 7fb38d7fa640 1 -- 172.21.15.98:0/3865480693 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb388001590 con 0x7fb39014a700 2024-03-29T11:49:27.011 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.000+0000 7fb38d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:49:27.011 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.000+0000 7fb38d7fa640 1 -- 172.21.15.98:0/3865480693 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fb3880019b0 con 0x7fb39014a700 2024-03-29T11:49:27.011 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.000+0000 7fb38d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:27.011 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.000+0000 7fb38d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:49:27.011 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.000+0000 7fb38d7fa640 10 monclient: dump: 2024-03-29T11:49:27.011 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:49:27.012 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:49:27.012 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:27.012 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:27.012 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:27.012 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:27.012 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:49:27.012 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:49:27.012 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:49:27.012 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:27.012 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.000+0000 7fb38effd640 4 set_mon_vals no callback set 2024-03-29T11:49:27.013 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.001+0000 7fb37affd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:27.014 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.001+0000 7fb37affd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:27.014 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.001+0000 7fb37affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:49:27.014 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.001+0000 7fb37affd640 1 -- 172.21.15.98:0/3865480693 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb390075f10 con 0x7fb39014a700 2024-03-29T11:49:27.014 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.002+0000 7fb38d7fa640 1 -- 172.21.15.98:0/3865480693 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7fb38801ae60 con 0x7fb39014a700 2024-03-29T11:49:27.014 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.003+0000 7fb38d7fa640 1 -- 172.21.15.98:0/3865480693 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7fb388084be0 con 0x7fb39014a700 2024-03-29T11:49:27.014 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.003+0000 7fb38d7fa640 10 client.4269.objecter ms_dispatch 0x7fb390000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:49:27.014 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.003+0000 7fb38d7fa640 3 client.4269.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:49:27.014 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.003+0000 7fb38d7fa640 3 client.4269.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:49:27.014 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.003+0000 7fb38d7fa640 20 client.4269.objecter dump_active .. 0 homeless 2024-03-29T11:49:27.014 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.004+0000 7fb394e92640 10 monclient: get_auth_request con 0x7fb37406dd60 auth_method 0 2024-03-29T11:49:27.014 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.006+0000 7fb38d7fa640 10 client.4269.objecter ms_handle_connect 0x7fb37406dd60 2024-03-29T11:49:27.017 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.011+0000 7fb38d7fa640 1 -- 172.21.15.98:0/3865480693 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb3880504c0 con 0x7fb39014a700 2024-03-29T11:49:27.017 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.011+0000 7fb38d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:27.017 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.011+0000 7fb38d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:49:27.341 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.335+0000 7fb37affd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:27.341 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.335+0000 7fb37affd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:27.341 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.335+0000 7fb37affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:49:27.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.335+0000 7fb37affd640 1 -- 172.21.15.98:0/3865480693 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb3901271f0 con 0x7fb39014a700 2024-03-29T11:49:27.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.336+0000 7fb38d7fa640 1 -- 172.21.15.98:0/3865480693 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7fb3880523f0 con 0x7fb39014a700 2024-03-29T11:49:27.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.336+0000 7fb38d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:27.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.336+0000 7fb38d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:49:27.343 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:49:27.343 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:49:27.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.339+0000 7fb37affd640 10 librados: watch_flush enter 2024-03-29T11:49:27.349 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.340+0000 7fb37affd640 10 librados: watch_flush exit 2024-03-29T11:49:27.349 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.340+0000 7fb37affd640 20 client.4269.objecter shutdown clearing up homeless session... 2024-03-29T11:49:27.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.340+0000 7fb37affd640 10 client.4269.objecter successfully canceled tick 2024-03-29T11:49:27.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.340+0000 7fb37affd640 1 -- 172.21.15.98:0/3865480693 >> v1:172.21.15.98:6816/4065186974 conn(0x7fb37406dd60 legacy=0x7fb374070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:27.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.340+0000 7fb37affd640 10 monclient: shutdown 2024-03-29T11:49:27.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.340+0000 7fb37affd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:49:27.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.340+0000 7fb37affd640 1 -- 172.21.15.98:0/3865480693 >> v1:172.21.15.98:6789/0 conn(0x7fb39014a700 legacy=0x7fb3901e6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:27.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.340+0000 7fb37affd640 1 -- 172.21.15.98:0/3865480693 shutdown_connections 2024-03-29T11:49:27.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.340+0000 7fb37affd640 1 -- 172.21.15.98:0/3865480693 >> 172.21.15.98:0/3865480693 conn(0x7fb39006b660 msgr2=0x7fb3900abe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:49:27.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.343+0000 7fb37affd640 1 -- 172.21.15.98:0/3865480693 shutdown_connections 2024-03-29T11:49:27.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.343+0000 7fb37affd640 1 -- 172.21.15.98:0/3865480693 wait complete. 2024-03-29T11:49:27.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:27.343+0000 7fb37affd640 1 librados: shutdown 2024-03-29T11:49:27.367 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:49:34.174 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:49:34.325 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.319+0000 7fba3e104640 10 monclient: get_monmap_and_config 2024-03-29T11:49:34.326 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.319+0000 7fba3e104640 10 monclient: build_initial_monmap 2024-03-29T11:49:34.326 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.319+0000 7fba3e104640 10 monclient: monmap: 2024-03-29T11:49:34.326 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:49:34.326 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:49:34.326 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:49:34.320747+0000 2024-03-29T11:49:34.326 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:49:34.320747+0000 2024-03-29T11:49:34.326 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:49:34.326 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:34.326 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:49:34.326 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:49:34.326 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:49:34.327 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:34.327 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.321+0000 7fba3e104640 1 Processor -- start 2024-03-29T11:49:34.328 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.322+0000 7fba3e104640 1 -- start start 2024-03-29T11:49:34.328 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.322+0000 7fba3e104640 10 monclient: init 2024-03-29T11:49:34.329 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.323+0000 7fba3e104640 10 monclient: _reopen_session rank -1 2024-03-29T11:49:34.329 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.323+0000 7fba3e104640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:49:34.329 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.323+0000 7fba3e104640 10 monclient(hunting): picked mon.noname-a con 0x7fba381457f0 addr v1:172.21.15.98:6789/0 2024-03-29T11:49:34.329 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.323+0000 7fba3e104640 10 monclient(hunting): picked mon.noname-c con 0x7fba3813f650 addr v1:172.21.15.121:6789/0 2024-03-29T11:49:34.329 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.323+0000 7fba3e104640 10 monclient(hunting): picked mon.noname-b con 0x7fba380a9d70 addr v1:172.21.15.98:6790/0 2024-03-29T11:49:34.329 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.323+0000 7fba3e104640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba380aaef0 con 0x7fba381457f0 2024-03-29T11:49:34.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.323+0000 7fba3e104640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba38145bd0 con 0x7fba3813f650 2024-03-29T11:49:34.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.323+0000 7fba3e104640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba38145eb0 con 0x7fba380a9d70 2024-03-29T11:49:34.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.323+0000 7fba3e104640 10 monclient(hunting): _renew_subs 2024-03-29T11:49:34.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.323+0000 7fba3e104640 10 monclient(hunting): authenticate will time out at 881.408278s 2024-03-29T11:49:34.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.323+0000 7fba377fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fba381457f0 0x7fba3813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48928/0 (socket says 172.21.15.98:48928) 2024-03-29T11:49:34.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.323+0000 7fba377fe640 1 -- 172.21.15.98:0/2269516169 learned_addr learned my addr 172.21.15.98:0/2269516169 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:49:34.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.323+0000 7fba37fff640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fba380a9d70 0x7fba3814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:59000/0 (socket says 172.21.15.98:59000) 2024-03-29T11:49:34.331 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.324+0000 7fba367fc640 1 -- 172.21.15.98:0/2269516169 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1093422366 0 0) 0x7fba38145bd0 con 0x7fba3813f650 2024-03-29T11:49:34.331 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.324+0000 7fba367fc640 10 monclient(hunting): _negotiate 2024-03-29T11:49:34.331 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.324+0000 7fba367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:34.331 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.324+0000 7fba367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:34.331 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.324+0000 7fba367fc640 10 monclient(hunting): my global_id is 4294 2024-03-29T11:49:34.331 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.324+0000 7fba367fc640 1 -- 172.21.15.98:0/2269516169 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba20003520 con 0x7fba3813f650 2024-03-29T11:49:34.331 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.324+0000 7fba367fc640 1 -- 172.21.15.98:0/2269516169 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 136676171 0 0) 0x7fba380aaef0 con 0x7fba381457f0 2024-03-29T11:49:34.331 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.324+0000 7fba367fc640 10 monclient(hunting): _negotiate 2024-03-29T11:49:34.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.324+0000 7fba367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:34.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.324+0000 7fba367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:34.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.324+0000 7fba367fc640 10 monclient(hunting): my global_id is 4272 2024-03-29T11:49:34.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.324+0000 7fba367fc640 1 -- 172.21.15.98:0/2269516169 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba38145bd0 con 0x7fba381457f0 2024-03-29T11:49:34.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.325+0000 7fba367fc640 1 -- 172.21.15.98:0/2269516169 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1699499874 0 0) 0x7fba20003520 con 0x7fba3813f650 2024-03-29T11:49:34.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.325+0000 7fba367fc640 1 -- 172.21.15.98:0/2269516169 >> v1:172.21.15.98:6790/0 conn(0x7fba380a9d70 legacy=0x7fba3814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:34.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.325+0000 7fba367fc640 1 -- 172.21.15.98:0/2269516169 >> v1:172.21.15.98:6789/0 conn(0x7fba381457f0 legacy=0x7fba3813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:34.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.325+0000 7fba367fc640 10 monclient: _finish_hunting 0 2024-03-29T11:49:34.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.325+0000 7fba367fc640 1 monclient: found mon.noname-c 2024-03-29T11:49:34.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.325+0000 7fba367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:49:34.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.325+0000 7fba367fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:49:34.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.325+0000 7fba367fc640 1 -- 172.21.15.98:0/2269516169 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba38146340 con 0x7fba3813f650 2024-03-29T11:49:34.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.325+0000 7fba367fc640 10 monclient: _finish_auth 0 2024-03-29T11:49:34.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.325+0000 7fba367fc640 10 monclient: _check_auth_tickets 2024-03-29T11:49:34.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.325+0000 7fba367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:49:34.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.325+0000 7fba367fc640 1 -- 172.21.15.98:0/2269516169 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fba28001690 con 0x7fba3813f650 2024-03-29T11:49:34.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.325+0000 7fba367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:34.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.325+0000 7fba367fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:49:34.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.325+0000 7fba367fc640 10 monclient: dump: 2024-03-29T11:49:34.333 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:49:34.333 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:49:34.333 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:34.333 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:34.333 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:34.333 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:34.333 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:49:34.333 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:49:34.333 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:49:34.334 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:34.334 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.325+0000 7fba3e104640 5 monclient: authenticate success, global_id 4294 2024-03-29T11:49:34.334 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.325+0000 7fba3e104640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:49:34.335 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.326+0000 7fba367fc640 1 -- 172.21.15.98:0/2269516169 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba28001870 con 0x7fba3813f650 2024-03-29T11:49:34.335 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.326+0000 7fba367fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:49:34.335 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.326+0000 7fba367fc640 1 -- 172.21.15.98:0/2269516169 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fba28001c80 con 0x7fba3813f650 2024-03-29T11:49:34.335 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.326+0000 7fba367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:34.336 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.326+0000 7fba367fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:49:34.336 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.326+0000 7fba367fc640 10 monclient: dump: 2024-03-29T11:49:34.336 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:49:34.336 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:49:34.336 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:34.336 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:34.336 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:34.336 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:34.336 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:49:34.336 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:49:34.336 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:49:34.336 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:34.336 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.326+0000 7fba3e104640 10 monclient: get_monmap_and_config success 2024-03-29T11:49:34.336 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.326+0000 7fba3e104640 4 set_mon_vals no callback set 2024-03-29T11:49:34.337 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.327+0000 7fba3e104640 10 monclient: shutdown 2024-03-29T11:49:34.337 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.327+0000 7fba3e104640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:49:34.337 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.327+0000 7fba3e104640 1 -- 172.21.15.98:0/2269516169 >> v1:172.21.15.121:6789/0 conn(0x7fba3813f650 legacy=0x7fba380b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:34.337 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.327+0000 7fba3e104640 1 -- 172.21.15.98:0/2269516169 shutdown_connections 2024-03-29T11:49:34.337 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.327+0000 7fba3e104640 1 -- 172.21.15.98:0/2269516169 >> 172.21.15.98:0/2269516169 conn(0x7fba38066a00 msgr2=0x7fba38066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:49:34.337 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.328+0000 7fba3e104640 1 -- 172.21.15.98:0/2269516169 shutdown_connections 2024-03-29T11:49:34.337 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.328+0000 7fba3e104640 1 -- 172.21.15.98:0/2269516169 wait complete. 2024-03-29T11:49:34.337 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.329+0000 7fba3e104640 10 monclient: build_initial_monmap 2024-03-29T11:49:34.337 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.329+0000 7fba3e104640 10 monclient: monmap: 2024-03-29T11:49:34.337 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:49:34.337 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:49:34.337 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:49:34.337 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:49:34.337 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:49:34.337 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:34.337 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:49:34.338 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:49:34.338 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:49:34.338 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:34.338 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.329+0000 7fba3e104640 1 librados: starting msgr at 2024-03-29T11:49:34.338 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.329+0000 7fba3e104640 1 librados: starting objecter 2024-03-29T11:49:34.338 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.329+0000 7fba3e104640 1 Processor -- start 2024-03-29T11:49:34.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.330+0000 7fba3e104640 1 -- start start 2024-03-29T11:49:34.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.330+0000 7fba3e104640 1 librados: setting wanted keys 2024-03-29T11:49:34.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.330+0000 7fba3e104640 1 librados: calling monclient init 2024-03-29T11:49:34.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.330+0000 7fba3e104640 10 monclient: init 2024-03-29T11:49:34.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.330+0000 7fba3e104640 10 monclient: _reopen_session rank -1 2024-03-29T11:49:34.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.330+0000 7fba3e104640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:49:34.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.331+0000 7fba3e104640 10 monclient(hunting): picked mon.noname-a con 0x7fba380a9d70 addr v1:172.21.15.98:6789/0 2024-03-29T11:49:34.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.331+0000 7fba3e104640 10 monclient(hunting): picked mon.noname-c con 0x7fba3813f650 addr v1:172.21.15.98:6790/0 2024-03-29T11:49:34.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.331+0000 7fba3e104640 10 monclient(hunting): picked mon.noname-b con 0x7fba381457f0 addr v1:172.21.15.121:6789/0 2024-03-29T11:49:34.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.331+0000 7fba3e104640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba3813b510 con 0x7fba380a9d70 2024-03-29T11:49:34.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.331+0000 7fba3e104640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba381cceb0 con 0x7fba381457f0 2024-03-29T11:49:34.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.331+0000 7fba3e104640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba381cd190 con 0x7fba3813f650 2024-03-29T11:49:34.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.331+0000 7fba3e104640 10 monclient(hunting): _renew_subs 2024-03-29T11:49:34.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.331+0000 7fba3e104640 10 monclient(hunting): authenticate will time out at 881.416497s 2024-03-29T11:49:34.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.331+0000 7fba36ffd640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fba3813f650 0x7fba38139090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:59166/0 (socket says 172.21.15.98:59166) 2024-03-29T11:49:34.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.331+0000 7fba36ffd640 1 -- 172.21.15.98:0/3057536411 learned_addr learned my addr 172.21.15.98:0/3057536411 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:49:34.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.331+0000 7fba1ffff640 10 client.?.objecter ms_handle_connect 0x7fba3813f650 2024-03-29T11:49:34.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.331+0000 7fba1ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:49:34.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.331+0000 7fba1ffff640 10 client.?.objecter ms_handle_connect 0x7fba381457f0 2024-03-29T11:49:34.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.331+0000 7fba1ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:49:34.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.332+0000 7fba1ffff640 10 client.?.objecter ms_handle_connect 0x7fba380a9d70 2024-03-29T11:49:34.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.332+0000 7fba1ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:49:34.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.332+0000 7fba1ffff640 1 -- 172.21.15.98:0/3057536411 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2515682696 0 0) 0x7fba381cd190 con 0x7fba3813f650 2024-03-29T11:49:34.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.332+0000 7fba1ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:49:34.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.332+0000 7fba1ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:34.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.332+0000 7fba1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:34.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.332+0000 7fba1ffff640 10 monclient(hunting): my global_id is 4277 2024-03-29T11:49:34.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.332+0000 7fba1ffff640 1 -- 172.21.15.98:0/3057536411 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba24004eb0 con 0x7fba3813f650 2024-03-29T11:49:34.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.332+0000 7fba1ffff640 1 -- 172.21.15.98:0/3057536411 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 43718314 0 0) 0x7fba381cceb0 con 0x7fba381457f0 2024-03-29T11:49:34.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.332+0000 7fba1ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:49:34.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.332+0000 7fba1ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:34.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.332+0000 7fba1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:34.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.332+0000 7fba1ffff640 10 monclient(hunting): my global_id is 4297 2024-03-29T11:49:34.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.332+0000 7fba1ffff640 1 -- 172.21.15.98:0/3057536411 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba381cd190 con 0x7fba381457f0 2024-03-29T11:49:34.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.333+0000 7fba1ffff640 1 -- 172.21.15.98:0/3057536411 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1758682216 0 0) 0x7fba3813b510 con 0x7fba380a9d70 2024-03-29T11:49:34.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.333+0000 7fba1ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:49:34.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.333+0000 7fba1ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:34.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.333+0000 7fba1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:34.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.333+0000 7fba1ffff640 10 monclient(hunting): my global_id is 4275 2024-03-29T11:49:34.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.333+0000 7fba1ffff640 1 -- 172.21.15.98:0/3057536411 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba24005360 con 0x7fba380a9d70 2024-03-29T11:49:34.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.333+0000 7fba1ffff640 1 -- 172.21.15.98:0/3057536411 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 500309484 0 0) 0x7fba381cd190 con 0x7fba381457f0 2024-03-29T11:49:34.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.333+0000 7fba1ffff640 1 -- 172.21.15.98:0/3057536411 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba3813b510 con 0x7fba381457f0 2024-03-29T11:49:34.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.333+0000 7fba1ffff640 1 -- 172.21.15.98:0/3057536411 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fba30001b20 con 0x7fba381457f0 2024-03-29T11:49:34.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.333+0000 7fba1ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:34.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.333+0000 7fba1ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:49:34.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.333+0000 7fba1ffff640 10 monclient(hunting): dump: 2024-03-29T11:49:34.345 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:49:34.345 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:49:34.345 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:34.345 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:34.346 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:34.346 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:34.346 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:49:34.346 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:49:34.346 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:49:34.346 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:34.346 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.333+0000 7fba1ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:49:34.346 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.334+0000 7fba1ffff640 1 -- 172.21.15.98:0/3057536411 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 739921647 0 0) 0x7fba3813b510 con 0x7fba381457f0 2024-03-29T11:49:34.346 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.334+0000 7fba1ffff640 1 -- 172.21.15.98:0/3057536411 >> v1:172.21.15.98:6790/0 conn(0x7fba3813f650 legacy=0x7fba38139090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:34.346 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.334+0000 7fba1ffff640 1 -- 172.21.15.98:0/3057536411 >> v1:172.21.15.98:6789/0 conn(0x7fba380a9d70 legacy=0x7fba3813b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:34.346 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.334+0000 7fba1ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:49:34.346 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.334+0000 7fba1ffff640 1 monclient: found mon.b 2024-03-29T11:49:34.346 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.334+0000 7fba1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:49:34.346 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.334+0000 7fba1ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:49:34.346 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.334+0000 7fba1ffff640 1 -- 172.21.15.98:0/3057536411 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba381cbc50 con 0x7fba381457f0 2024-03-29T11:49:34.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.334+0000 7fba1ffff640 10 monclient: _finish_auth 0 2024-03-29T11:49:34.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.334+0000 7fba1ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:49:34.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.334+0000 7fba1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:49:34.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.334+0000 7fba3e104640 5 monclient: authenticate success, global_id 4297 2024-03-29T11:49:34.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.334+0000 7fba3e104640 10 monclient: _renew_subs 2024-03-29T11:49:34.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.334+0000 7fba3e104640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:49:34.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.334+0000 7fba3e104640 1 -- 172.21.15.98:0/3057536411 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba381ce5c0 con 0x7fba381457f0 2024-03-29T11:49:34.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.334+0000 7fba3e104640 10 client.4297.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:49:34.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.334+0000 7fba3e104640 10 monclient: _renew_subs 2024-03-29T11:49:34.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.334+0000 7fba3e104640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:49:34.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.334+0000 7fba3e104640 1 -- 172.21.15.98:0/3057536411 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fba381ce8c0 con 0x7fba381457f0 2024-03-29T11:49:34.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.334+0000 7fba3e104640 1 librados: init done 2024-03-29T11:49:34.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.335+0000 7fba1ffff640 1 -- 172.21.15.98:0/3057536411 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba30001ff0 con 0x7fba381457f0 2024-03-29T11:49:34.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.335+0000 7fba1ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:49:34.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.335+0000 7fba1ffff640 1 -- 172.21.15.98:0/3057536411 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fba30001b20 con 0x7fba381457f0 2024-03-29T11:49:34.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.335+0000 7fba1ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:34.348 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.335+0000 7fba357fa640 4 set_mon_vals no callback set 2024-03-29T11:49:34.348 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.335+0000 7fba1ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:49:34.348 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.335+0000 7fba1ffff640 10 monclient: dump: 2024-03-29T11:49:34.348 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:49:34.348 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:49:34.348 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:34.348 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:34.348 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:34.348 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:34.348 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:49:34.348 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:49:34.348 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:49:34.348 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:34.349 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.337+0000 7fba1ffff640 1 -- 172.21.15.98:0/3057536411 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7fba3001a9d0 con 0x7fba381457f0 2024-03-29T11:49:34.349 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.337+0000 7fba3e104640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:34.349 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.337+0000 7fba3e104640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:34.349 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.337+0000 7fba3e104640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:49:34.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.337+0000 7fba3e104640 1 -- 172.21.15.98:0/3057536411 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba18005080 con 0x7fba381457f0 2024-03-29T11:49:34.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.338+0000 7fba377fe640 10 monclient: get_auth_request con 0x7fba2406e390 auth_method 0 2024-03-29T11:49:34.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.338+0000 7fba1ffff640 1 -- 172.21.15.98:0/3057536411 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7fba30084610 con 0x7fba381457f0 2024-03-29T11:49:34.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.338+0000 7fba1ffff640 10 client.4297.objecter ms_dispatch 0x7fba38000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:49:34.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.338+0000 7fba1ffff640 3 client.4297.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:49:34.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.338+0000 7fba1ffff640 3 client.4297.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:49:34.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.338+0000 7fba1ffff640 20 client.4297.objecter dump_active .. 0 homeless 2024-03-29T11:49:34.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.338+0000 7fba1ffff640 10 client.4297.objecter ms_handle_connect 0x7fba2406e390 2024-03-29T11:49:34.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.344+0000 7fba1ffff640 1 -- 172.21.15.98:0/3057536411 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fba3004fe60 con 0x7fba381457f0 2024-03-29T11:49:34.351 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.344+0000 7fba1ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:34.351 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.344+0000 7fba1ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:49:34.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.723+0000 7fba3e104640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:34.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.723+0000 7fba3e104640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:34.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.723+0000 7fba3e104640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:49:34.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.723+0000 7fba3e104640 1 -- 172.21.15.98:0/3057536411 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fba18005330 con 0x7fba381457f0 2024-03-29T11:49:34.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.724+0000 7fba1ffff640 1 -- 172.21.15.98:0/3057536411 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7fba3004fe60 con 0x7fba381457f0 2024-03-29T11:49:34.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.724+0000 7fba1ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:34.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.724+0000 7fba1ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:49:34.731 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:49:34.731 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:49:34.735 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.729+0000 7fba3e104640 10 librados: watch_flush enter 2024-03-29T11:49:34.735 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.729+0000 7fba3e104640 10 librados: watch_flush exit 2024-03-29T11:49:34.736 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.729+0000 7fba3e104640 20 client.4297.objecter shutdown clearing up homeless session... 2024-03-29T11:49:34.736 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.729+0000 7fba3e104640 10 client.4297.objecter successfully canceled tick 2024-03-29T11:49:34.736 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.729+0000 7fba3e104640 1 -- 172.21.15.98:0/3057536411 >> v1:172.21.15.98:6816/4065186974 conn(0x7fba2406e390 legacy=0x7fba24070850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:34.736 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.729+0000 7fba3e104640 10 monclient: shutdown 2024-03-29T11:49:34.736 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.729+0000 7fba3e104640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:49:34.736 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.729+0000 7fba3e104640 1 -- 172.21.15.98:0/3057536411 >> v1:172.21.15.121:6789/0 conn(0x7fba381457f0 legacy=0x7fba381397a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:34.736 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.729+0000 7fba3e104640 1 -- 172.21.15.98:0/3057536411 shutdown_connections 2024-03-29T11:49:34.736 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.729+0000 7fba3e104640 1 -- 172.21.15.98:0/3057536411 >> 172.21.15.98:0/3057536411 conn(0x7fba38066a00 msgr2=0x7fba380b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:49:34.736 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.729+0000 7fba3e104640 1 -- 172.21.15.98:0/3057536411 shutdown_connections 2024-03-29T11:49:34.736 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.730+0000 7fba3e104640 1 -- 172.21.15.98:0/3057536411 wait complete. 2024-03-29T11:49:34.736 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:34.730+0000 7fba3e104640 1 librados: shutdown 2024-03-29T11:49:34.752 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:49:41.558 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:49:41.705 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.699+0000 7f4b1e3c8640 10 monclient: get_monmap_and_config 2024-03-29T11:49:41.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.699+0000 7f4b1e3c8640 10 monclient: build_initial_monmap 2024-03-29T11:49:41.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.699+0000 7f4b1e3c8640 10 monclient: monmap: 2024-03-29T11:49:41.706 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:49:41.706 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:49:41.706 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:49:41.700409+0000 2024-03-29T11:49:41.706 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:49:41.700409+0000 2024-03-29T11:49:41.706 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:49:41.706 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:41.706 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:49:41.706 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:49:41.706 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:49:41.706 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:41.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.700+0000 7f4b1e3c8640 1 Processor -- start 2024-03-29T11:49:41.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.700+0000 7f4b1e3c8640 1 -- start start 2024-03-29T11:49:41.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.700+0000 7f4b1e3c8640 10 monclient: init 2024-03-29T11:49:41.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.701+0000 7f4b1e3c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:49:41.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.701+0000 7f4b1e3c8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:49:41.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.701+0000 7f4b1e3c8640 10 monclient(hunting): picked mon.noname-b con 0x7f4b181422b0 addr v1:172.21.15.98:6790/0 2024-03-29T11:49:41.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.701+0000 7f4b1e3c8640 10 monclient(hunting): picked mon.noname-a con 0x7f4b18142690 addr v1:172.21.15.98:6789/0 2024-03-29T11:49:41.709 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.701+0000 7f4b1e3c8640 10 monclient(hunting): picked mon.noname-c con 0x7f4b180b17d0 addr v1:172.21.15.121:6789/0 2024-03-29T11:49:41.709 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.701+0000 7f4b1e3c8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b18142c80 con 0x7f4b18142690 2024-03-29T11:49:41.709 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.701+0000 7f4b1e3c8640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b180aae40 con 0x7f4b180b17d0 2024-03-29T11:49:41.709 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.701+0000 7f4b1e3c8640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b1813fbd0 con 0x7f4b181422b0 2024-03-29T11:49:41.709 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.701+0000 7f4b1e3c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:49:41.709 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.701+0000 7f4b1e3c8640 10 monclient(hunting): authenticate will time out at 888.786495s 2024-03-29T11:49:41.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.703+0000 7f4b17fff640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f4b181422b0 0x7f4b1813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:55852/0 (socket says 172.21.15.98:55852) 2024-03-29T11:49:41.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.703+0000 7f4b17fff640 1 -- 172.21.15.98:0/287258196 learned_addr learned my addr 172.21.15.98:0/287258196 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:49:41.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.704+0000 7f4b16ffd640 1 -- 172.21.15.98:0/287258196 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 516245796 0 0) 0x7f4b180aae40 con 0x7f4b180b17d0 2024-03-29T11:49:41.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.704+0000 7f4b16ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:49:41.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.704+0000 7f4b16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:41.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.704+0000 7f4b16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:41.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.704+0000 7f4b16ffd640 10 monclient(hunting): my global_id is 4300 2024-03-29T11:49:41.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.704+0000 7f4b16ffd640 1 -- 172.21.15.98:0/287258196 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b00003520 con 0x7f4b180b17d0 2024-03-29T11:49:41.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.704+0000 7f4b16ffd640 1 -- 172.21.15.98:0/287258196 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1063910146 0 0) 0x7f4b00003520 con 0x7f4b180b17d0 2024-03-29T11:49:41.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.704+0000 7f4b16ffd640 1 -- 172.21.15.98:0/287258196 >> v1:172.21.15.98:6790/0 conn(0x7f4b181422b0 legacy=0x7f4b1813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:41.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.704+0000 7f4b16ffd640 1 -- 172.21.15.98:0/287258196 >> v1:172.21.15.98:6789/0 conn(0x7f4b18142690 legacy=0x7f4b180b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:49:41.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.704+0000 7f4b16ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:49:41.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.704+0000 7f4b16ffd640 1 monclient: found mon.noname-c 2024-03-29T11:49:41.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.704+0000 7f4b16ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:49:41.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.704+0000 7f4b16ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:49:41.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.704+0000 7f4b16ffd640 1 -- 172.21.15.98:0/287258196 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b18151940 con 0x7f4b180b17d0 2024-03-29T11:49:41.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.704+0000 7f4b16ffd640 10 monclient: _finish_auth 0 2024-03-29T11:49:41.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.704+0000 7f4b16ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:49:41.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.704+0000 7f4b16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:49:41.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.704+0000 7f4b1e3c8640 5 monclient: authenticate success, global_id 4300 2024-03-29T11:49:41.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.704+0000 7f4b1e3c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:49:41.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.704+0000 7f4b16ffd640 1 -- 172.21.15.98:0/287258196 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f4b10001670 con 0x7f4b180b17d0 2024-03-29T11:49:41.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.704+0000 7f4b16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:41.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.704+0000 7f4b16ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:49:41.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.704+0000 7f4b16ffd640 10 monclient: dump: 2024-03-29T11:49:41.713 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:49:41.713 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:49:41.713 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:41.713 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:41.714 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:41.714 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:41.714 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:49:41.714 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:49:41.714 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:49:41.714 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:41.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.704+0000 7f4b1e3c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:49:41.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.704+0000 7f4b16ffd640 1 -- 172.21.15.98:0/287258196 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b100018d0 con 0x7f4b180b17d0 2024-03-29T11:49:41.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.704+0000 7f4b16ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:49:41.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.704+0000 7f4b16ffd640 1 -- 172.21.15.98:0/287258196 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f4b10001c60 con 0x7f4b180b17d0 2024-03-29T11:49:41.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.704+0000 7f4b1e3c8640 10 monclient: get_monmap_and_config success 2024-03-29T11:49:41.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.704+0000 7f4b1e3c8640 4 set_mon_vals no callback set 2024-03-29T11:49:41.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.705+0000 7f4b16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:41.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.705+0000 7f4b16ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:49:41.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.705+0000 7f4b16ffd640 10 monclient: dump: 2024-03-29T11:49:41.715 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:49:41.715 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:49:41.715 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:41.715 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:41.715 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:41.715 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:41.715 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:49:41.715 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:49:41.715 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:49:41.715 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:41.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.705+0000 7f4b1e3c8640 10 monclient: shutdown 2024-03-29T11:49:41.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.705+0000 7f4b1e3c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:49:41.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.705+0000 7f4b1e3c8640 1 -- 172.21.15.98:0/287258196 >> v1:172.21.15.121:6789/0 conn(0x7f4b180b17d0 legacy=0x7f4b180b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:41.717 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.706+0000 7f4b1e3c8640 1 -- 172.21.15.98:0/287258196 shutdown_connections 2024-03-29T11:49:41.717 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.706+0000 7f4b1e3c8640 1 -- 172.21.15.98:0/287258196 >> 172.21.15.98:0/287258196 conn(0x7f4b18066a00 msgr2=0x7f4b18066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:49:41.717 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.706+0000 7f4b1e3c8640 1 -- 172.21.15.98:0/287258196 shutdown_connections 2024-03-29T11:49:41.717 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.707+0000 7f4b1e3c8640 1 -- 172.21.15.98:0/287258196 wait complete. 2024-03-29T11:49:41.717 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.708+0000 7f4b1e3c8640 10 monclient: build_initial_monmap 2024-03-29T11:49:41.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.708+0000 7f4b1e3c8640 10 monclient: monmap: 2024-03-29T11:49:41.718 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:49:41.718 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:49:41.718 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:49:41.718 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:49:41.718 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:49:41.718 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:41.718 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:49:41.718 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:49:41.718 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:49:41.718 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:41.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.708+0000 7f4b1e3c8640 1 librados: starting msgr at 2024-03-29T11:49:41.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.708+0000 7f4b1e3c8640 1 librados: starting objecter 2024-03-29T11:49:41.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.709+0000 7f4b1e3c8640 1 Processor -- start 2024-03-29T11:49:41.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.709+0000 7f4b1e3c8640 1 -- start start 2024-03-29T11:49:41.719 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.709+0000 7f4b1e3c8640 1 librados: setting wanted keys 2024-03-29T11:49:41.719 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.709+0000 7f4b1e3c8640 1 librados: calling monclient init 2024-03-29T11:49:41.719 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.709+0000 7f4b1e3c8640 10 monclient: init 2024-03-29T11:49:41.719 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.710+0000 7f4b1e3c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:49:41.719 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.710+0000 7f4b1e3c8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:49:41.719 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.710+0000 7f4b1e3c8640 10 monclient(hunting): picked mon.noname-b con 0x7f4b180b17d0 addr v1:172.21.15.121:6789/0 2024-03-29T11:49:41.719 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.710+0000 7f4b1e3c8640 10 monclient(hunting): picked mon.noname-a con 0x7f4b181422b0 addr v1:172.21.15.98:6789/0 2024-03-29T11:49:41.719 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.710+0000 7f4b1e3c8640 10 monclient(hunting): picked mon.noname-c con 0x7f4b18142690 addr v1:172.21.15.98:6790/0 2024-03-29T11:49:41.719 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.710+0000 7f4b1e3c8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b180abab0 con 0x7f4b181422b0 2024-03-29T11:49:41.719 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.710+0000 7f4b1e3c8640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b180abce0 con 0x7f4b180b17d0 2024-03-29T11:49:41.719 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.710+0000 7f4b1e3c8640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b180b7dc0 con 0x7f4b18142690 2024-03-29T11:49:41.719 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.710+0000 7f4b1e3c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:49:41.719 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.710+0000 7f4b1e3c8640 10 monclient(hunting): authenticate will time out at 888.795268s 2024-03-29T11:49:41.725 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.713+0000 7f4b17fff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f4b180b17d0 0x7f4b181da890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:46199/0 (socket says 172.21.15.98:46199) 2024-03-29T11:49:41.725 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.713+0000 7f4b17fff640 1 -- 172.21.15.98:0/4168901652 learned_addr learned my addr 172.21.15.98:0/4168901652 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:49:41.725 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.713+0000 7f4b14ff9640 10 client.?.objecter ms_handle_connect 0x7f4b180b17d0 2024-03-29T11:49:41.725 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.713+0000 7f4b17fff640 1 -- 172.21.15.98:0/4168901652 >> v1:172.21.15.121:6789/0 conn(0x7f4b180b17d0 legacy=0x7f4b181da890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-29T11:49:41.725 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.713+0000 7f4b17fff640 1 -- 172.21.15.98:0/4168901652 >> v1:172.21.15.121:6789/0 conn(0x7f4b180b17d0 legacy=0x7f4b181da890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:49:41.725 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.713+0000 7f4b17fff640 1 --1- 172.21.15.98:0/4168901652 >> v1:172.21.15.121:6789/0 conn(0x7f4b180b17d0 0x7f4b181da890 :-1 s=OPENED pgs=189922 cs=1 l=1).handle_message read tag failed 2024-03-29T11:49:41.725 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.713+0000 7f4b17fff640 1 --1- 172.21.15.98:0/4168901652 >> v1:172.21.15.121:6789/0 conn(0x7f4b180b17d0 0x7f4b181da890 :-1 s=OPENED pgs=189922 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:49:41.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.713+0000 7f4b14ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:49:41.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.713+0000 7f4b14ff9640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.121:6789/0 2024-03-29T11:49:41.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.714+0000 7f4b14ff9640 10 client.?.objecter ms_handle_connect 0x7f4b181422b0 2024-03-29T11:49:41.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.714+0000 7f4b14ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:49:41.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.714+0000 7f4b14ff9640 1 -- 172.21.15.98:0/4168901652 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2750677501 0 0) 0x7f4b180abab0 con 0x7f4b181422b0 2024-03-29T11:49:41.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.714+0000 7f4b14ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:49:41.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.714+0000 7f4b14ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:41.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.714+0000 7f4b14ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:41.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.714+0000 7f4b14ff9640 10 monclient(hunting): my global_id is 4278 2024-03-29T11:49:41.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.714+0000 7f4b14ff9640 1 -- 172.21.15.98:0/4168901652 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b04004eb0 con 0x7f4b181422b0 2024-03-29T11:49:41.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.715+0000 7f4b14ff9640 1 -- 172.21.15.98:0/4168901652 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4258572981 0 0) 0x7f4b04004eb0 con 0x7f4b181422b0 2024-03-29T11:49:41.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.715+0000 7f4b14ff9640 1 -- 172.21.15.98:0/4168901652 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b180abab0 con 0x7f4b181422b0 2024-03-29T11:49:41.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.715+0000 7f4b14ff9640 10 client.?.objecter ms_handle_connect 0x7f4b18142690 2024-03-29T11:49:41.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.715+0000 7f4b14ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:49:41.727 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.715+0000 7f4b14ff9640 1 -- 172.21.15.98:0/4168901652 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f4b0c0017d0 con 0x7f4b181422b0 2024-03-29T11:49:41.727 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.715+0000 7f4b14ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:41.727 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.715+0000 7f4b14ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:49:41.727 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.715+0000 7f4b14ff9640 10 monclient(hunting): dump: 2024-03-29T11:49:41.727 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:49:41.727 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:49:41.727 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:41.727 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:41.727 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:41.727 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:41.727 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:49:41.727 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:49:41.727 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:49:41.727 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:41.727 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.715+0000 7f4b14ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:49:41.727 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.716+0000 7f4b14ff9640 1 -- 172.21.15.98:0/4168901652 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2744519828 0 0) 0x7f4b180b7dc0 con 0x7f4b18142690 2024-03-29T11:49:41.728 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.716+0000 7f4b14ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:49:41.728 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.716+0000 7f4b14ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:41.728 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.716+0000 7f4b14ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:41.728 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.716+0000 7f4b14ff9640 10 monclient(hunting): my global_id is 4283 2024-03-29T11:49:41.728 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.716+0000 7f4b14ff9640 1 -- 172.21.15.98:0/4168901652 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b04004eb0 con 0x7f4b18142690 2024-03-29T11:49:41.728 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.716+0000 7f4b14ff9640 1 -- 172.21.15.98:0/4168901652 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1191863075 0 0) 0x7f4b180abab0 con 0x7f4b181422b0 2024-03-29T11:49:41.728 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.716+0000 7f4b14ff9640 1 -- 172.21.15.98:0/4168901652 >> v1:172.21.15.98:6790/0 conn(0x7f4b18142690 legacy=0x7f4b181e2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:41.728 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.716+0000 7f4b14ff9640 1 -- 172.21.15.98:0/4168901652 >> v1:172.21.15.121:6789/0 conn(0x7f4b180b17d0 legacy=0x7f4b181da890 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:49:41.728 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.716+0000 7f4b14ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:49:41.728 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.716+0000 7f4b14ff9640 1 monclient: found mon.a 2024-03-29T11:49:41.728 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.716+0000 7f4b14ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:49:41.728 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.716+0000 7f4b14ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:49:41.728 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.716+0000 7f4b14ff9640 1 -- 172.21.15.98:0/4168901652 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b180b7480 con 0x7f4b181422b0 2024-03-29T11:49:41.728 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.716+0000 7f4b14ff9640 10 monclient: _finish_auth 0 2024-03-29T11:49:41.728 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.716+0000 7f4b14ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:49:41.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.716+0000 7f4b14ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:49:41.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.716+0000 7f4b1e3c8640 5 monclient: authenticate success, global_id 4278 2024-03-29T11:49:41.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.716+0000 7f4b1e3c8640 10 monclient: _renew_subs 2024-03-29T11:49:41.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.716+0000 7f4b1e3c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:49:41.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.717+0000 7f4b1e3c8640 1 -- 172.21.15.98:0/4168901652 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b181dafa0 con 0x7f4b181422b0 2024-03-29T11:49:41.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.717+0000 7f4b1e3c8640 10 client.4278.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:49:41.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.717+0000 7f4b1e3c8640 10 monclient: _renew_subs 2024-03-29T11:49:41.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.717+0000 7f4b1e3c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:49:41.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.717+0000 7f4b1e3c8640 1 -- 172.21.15.98:0/4168901652 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b180b84c0 con 0x7f4b181422b0 2024-03-29T11:49:41.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.717+0000 7f4b1e3c8640 1 librados: init done 2024-03-29T11:49:41.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.717+0000 7f4b14ff9640 1 -- 172.21.15.98:0/4168901652 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b0c001930 con 0x7f4b181422b0 2024-03-29T11:49:41.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.717+0000 7f4b14ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:49:41.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.718+0000 7f4b1e3c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:41.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.718+0000 7f4b14ff9640 1 -- 172.21.15.98:0/4168901652 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f4b0c003790 con 0x7f4b181422b0 2024-03-29T11:49:41.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.718+0000 7f4b1e3c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:41.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.718+0000 7f4b1e3c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:49:41.731 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.718+0000 7f4b167fc640 4 set_mon_vals no callback set 2024-03-29T11:49:41.731 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.718+0000 7f4b1e3c8640 1 -- 172.21.15.98:0/4168901652 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b180b7f50 con 0x7f4b181422b0 2024-03-29T11:49:41.731 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.720+0000 7f4b14ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:41.731 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.720+0000 7f4b14ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:49:41.731 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.720+0000 7f4b14ff9640 10 monclient: dump: 2024-03-29T11:49:41.731 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:49:41.731 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:49:41.731 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:41.732 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:41.732 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:41.732 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:41.732 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:49:41.732 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:49:41.732 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:49:41.732 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:41.732 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.721+0000 7f4b14ff9640 1 -- 172.21.15.98:0/4168901652 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f4b0c019760 con 0x7f4b181422b0 2024-03-29T11:49:41.732 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.721+0000 7f4b17fff640 10 monclient: get_auth_request con 0x7f4b0406dd10 auth_method 0 2024-03-29T11:49:41.732 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.722+0000 7f4b14ff9640 10 client.4278.objecter ms_handle_connect 0x7f4b0406dd10 2024-03-29T11:49:41.732 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.722+0000 7f4b14ff9640 1 -- 172.21.15.98:0/4168901652 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f4b0c083700 con 0x7f4b181422b0 2024-03-29T11:49:41.732 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.722+0000 7f4b14ff9640 10 client.4278.objecter ms_dispatch 0x7f4b18000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:49:41.732 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.722+0000 7f4b14ff9640 3 client.4278.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:49:41.732 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.722+0000 7f4b14ff9640 3 client.4278.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:49:41.732 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.722+0000 7f4b14ff9640 20 client.4278.objecter dump_active .. 0 homeless 2024-03-29T11:49:41.736 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.730+0000 7f4b14ff9640 1 -- 172.21.15.98:0/4168901652 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4b0c04efe0 con 0x7f4b181422b0 2024-03-29T11:49:41.736 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.730+0000 7f4b14ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:41.736 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:41.730+0000 7f4b14ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:49:42.059 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:42.053+0000 7f4b1e3c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:42.060 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:42.053+0000 7f4b1e3c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:42.060 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:42.053+0000 7f4b1e3c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:49:42.060 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:42.053+0000 7f4b1e3c8640 1 -- 172.21.15.98:0/4168901652 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b18075f10 con 0x7f4b181422b0 2024-03-29T11:49:42.061 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:42.055+0000 7f4b14ff9640 1 -- 172.21.15.98:0/4168901652 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f4b0c050f10 con 0x7f4b181422b0 2024-03-29T11:49:42.061 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:42.055+0000 7f4b14ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:42.061 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:42.055+0000 7f4b14ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:49:42.061 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:49:42.061 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:49:42.064 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:42.058+0000 7f4b1e3c8640 10 librados: watch_flush enter 2024-03-29T11:49:42.064 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:42.058+0000 7f4b1e3c8640 10 librados: watch_flush exit 2024-03-29T11:49:42.064 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:42.058+0000 7f4b1e3c8640 20 client.4278.objecter shutdown clearing up homeless session... 2024-03-29T11:49:42.065 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:42.058+0000 7f4b1e3c8640 10 client.4278.objecter successfully canceled tick 2024-03-29T11:49:42.065 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:42.058+0000 7f4b1e3c8640 1 -- 172.21.15.98:0/4168901652 >> v1:172.21.15.98:6816/4065186974 conn(0x7f4b0406dd10 legacy=0x7f4b040701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:42.065 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:42.058+0000 7f4b1e3c8640 10 monclient: shutdown 2024-03-29T11:49:42.065 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:42.058+0000 7f4b1e3c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:49:42.065 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:42.058+0000 7f4b1e3c8640 1 -- 172.21.15.98:0/4168901652 >> v1:172.21.15.98:6789/0 conn(0x7f4b181422b0 legacy=0x7f4b1814da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:42.066 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:42.060+0000 7f4b1e3c8640 1 -- 172.21.15.98:0/4168901652 shutdown_connections 2024-03-29T11:49:42.066 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:42.060+0000 7f4b1e3c8640 1 -- 172.21.15.98:0/4168901652 >> 172.21.15.98:0/4168901652 conn(0x7f4b18066a00 msgr2=0x7f4b180b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:49:42.067 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:42.061+0000 7f4b1e3c8640 1 -- 172.21.15.98:0/4168901652 shutdown_connections 2024-03-29T11:49:42.067 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:42.061+0000 7f4b1e3c8640 1 -- 172.21.15.98:0/4168901652 wait complete. 2024-03-29T11:49:42.067 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:42.061+0000 7f4b1e3c8640 1 librados: shutdown 2024-03-29T11:49:42.085 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:49:48.891 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:49:49.029 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.022+0000 7f7df4bbc640 10 monclient: get_monmap_and_config 2024-03-29T11:49:49.029 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.022+0000 7f7df4bbc640 10 monclient: build_initial_monmap 2024-03-29T11:49:49.029 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.022+0000 7f7df4bbc640 10 monclient: monmap: 2024-03-29T11:49:49.030 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:49:49.030 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:49:49.030 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:49:49.023264+0000 2024-03-29T11:49:49.030 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:49:49.023264+0000 2024-03-29T11:49:49.030 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:49:49.030 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:49.030 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:49:49.030 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:49:49.030 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:49:49.030 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:49.030 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.023+0000 7f7df4bbc640 1 Processor -- start 2024-03-29T11:49:49.030 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.023+0000 7f7df4bbc640 1 -- start start 2024-03-29T11:49:49.030 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.023+0000 7f7df4bbc640 10 monclient: init 2024-03-29T11:49:49.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.024+0000 7f7df4bbc640 10 monclient: _reopen_session rank -1 2024-03-29T11:49:49.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.024+0000 7f7df4bbc640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:49:49.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.024+0000 7f7df4bbc640 10 monclient(hunting): picked mon.noname-c con 0x7f7df00a9750 addr v1:172.21.15.121:6789/0 2024-03-29T11:49:49.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.024+0000 7f7df4bbc640 10 monclient(hunting): picked mon.noname-b con 0x7f7df0149ac0 addr v1:172.21.15.98:6790/0 2024-03-29T11:49:49.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.024+0000 7f7df4bbc640 10 monclient(hunting): picked mon.noname-a con 0x7f7df014fe50 addr v1:172.21.15.98:6789/0 2024-03-29T11:49:49.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.024+0000 7f7df4bbc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7df00aa4a0 con 0x7f7df014fe50 2024-03-29T11:49:49.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.024+0000 7f7df4bbc640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7df014a680 con 0x7f7df00a9750 2024-03-29T11:49:49.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.024+0000 7f7df4bbc640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7df00af290 con 0x7f7df0149ac0 2024-03-29T11:49:49.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.024+0000 7f7df4bbc640 10 monclient(hunting): _renew_subs 2024-03-29T11:49:49.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.024+0000 7f7df4bbc640 10 monclient(hunting): authenticate will time out at 896.109674s 2024-03-29T11:49:49.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.024+0000 7f7dee59c640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f7df00a9750 0x7f7df01436f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:33284/0 (socket says 172.21.15.98:33284) 2024-03-29T11:49:49.033 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.024+0000 7f7dedd9b640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f7df0149ac0 0x7f7df0149f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:58514/0 (socket says 172.21.15.98:58514) 2024-03-29T11:49:49.033 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.024+0000 7f7dee59c640 1 -- 172.21.15.98:0/4145638323 learned_addr learned my addr 172.21.15.98:0/4145638323 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:49:49.033 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.025+0000 7f7ded59a640 1 -- 172.21.15.98:0/4145638323 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2042239937 0 0) 0x7f7df00af290 con 0x7f7df0149ac0 2024-03-29T11:49:49.033 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.025+0000 7f7ded59a640 10 monclient(hunting): _negotiate 2024-03-29T11:49:49.033 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.025+0000 7f7ded59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:49.033 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.025+0000 7f7ded59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:49.033 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.025+0000 7f7ded59a640 10 monclient(hunting): my global_id is 4286 2024-03-29T11:49:49.033 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.025+0000 7f7ded59a640 1 -- 172.21.15.98:0/4145638323 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dd8003520 con 0x7f7df0149ac0 2024-03-29T11:49:49.033 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.025+0000 7f7ded59a640 1 -- 172.21.15.98:0/4145638323 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 856622397 0 0) 0x7f7df014a680 con 0x7f7df00a9750 2024-03-29T11:49:49.033 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.025+0000 7f7ded59a640 10 monclient(hunting): _negotiate 2024-03-29T11:49:49.033 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.025+0000 7f7ded59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:49.033 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.025+0000 7f7ded59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:49.033 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.025+0000 7f7ded59a640 10 monclient(hunting): my global_id is 4303 2024-03-29T11:49:49.034 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.025+0000 7f7ded59a640 1 -- 172.21.15.98:0/4145638323 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7df00af290 con 0x7f7df00a9750 2024-03-29T11:49:49.034 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.025+0000 7f7ded59a640 1 -- 172.21.15.98:0/4145638323 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4286888463 0 0) 0x7f7dd8003520 con 0x7f7df0149ac0 2024-03-29T11:49:49.034 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.025+0000 7f7ded59a640 1 -- 172.21.15.98:0/4145638323 >> v1:172.21.15.121:6789/0 conn(0x7f7df00a9750 legacy=0x7f7df01436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:49.034 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.025+0000 7f7ded59a640 1 -- 172.21.15.98:0/4145638323 >> v1:172.21.15.98:6789/0 conn(0x7f7df014fe50 legacy=0x7f7df0152240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:49:49.034 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.028+0000 7f7ded59a640 10 monclient: _finish_hunting 0 2024-03-29T11:49:49.035 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.028+0000 7f7ded59a640 1 monclient: found mon.noname-b 2024-03-29T11:49:49.035 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.028+0000 7f7ded59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:49:49.035 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.028+0000 7f7ded59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:49:49.035 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.028+0000 7f7ded59a640 1 -- 172.21.15.98:0/4145638323 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df00afb40 con 0x7f7df0149ac0 2024-03-29T11:49:49.035 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.028+0000 7f7ded59a640 10 monclient: _finish_auth 0 2024-03-29T11:49:49.035 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.028+0000 7f7ded59a640 10 monclient: _check_auth_tickets 2024-03-29T11:49:49.035 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.028+0000 7f7ded59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:49:49.035 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.028+0000 7f7ded59a640 1 -- 172.21.15.98:0/4145638323 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f7de4001690 con 0x7f7df0149ac0 2024-03-29T11:49:49.035 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.028+0000 7f7ded59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:49.035 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.028+0000 7f7ded59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:49:49.035 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.028+0000 7f7ded59a640 10 monclient: dump: 2024-03-29T11:49:49.035 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:49:49.035 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:49:49.035 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:49.035 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:49.036 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:49.036 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:49.036 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:49:49.036 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:49:49.036 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:49:49.036 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:49.036 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.028+0000 7f7df4bbc640 5 monclient: authenticate success, global_id 4286 2024-03-29T11:49:49.036 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.028+0000 7f7df4bbc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:49:49.036 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.028+0000 7f7ded59a640 1 -- 172.21.15.98:0/4145638323 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7de4001870 con 0x7f7df0149ac0 2024-03-29T11:49:49.036 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.028+0000 7f7ded59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:49:49.036 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.028+0000 7f7df4bbc640 10 monclient: get_monmap_and_config success 2024-03-29T11:49:49.036 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.028+0000 7f7df4bbc640 4 set_mon_vals no callback set 2024-03-29T11:49:49.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.029+0000 7f7df4bbc640 10 monclient: shutdown 2024-03-29T11:49:49.038 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.029+0000 7f7df4bbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:49:49.038 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.029+0000 7f7df4bbc640 1 -- 172.21.15.98:0/4145638323 >> v1:172.21.15.98:6790/0 conn(0x7f7df0149ac0 legacy=0x7f7df0149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:49.038 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.029+0000 7f7df4bbc640 1 -- 172.21.15.98:0/4145638323 shutdown_connections 2024-03-29T11:49:49.038 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.029+0000 7f7df4bbc640 1 -- 172.21.15.98:0/4145638323 >> 172.21.15.98:0/4145638323 conn(0x7f7df00b35e0 msgr2=0x7f7df00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:49:49.038 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.029+0000 7f7df4bbc640 1 -- 172.21.15.98:0/4145638323 shutdown_connections 2024-03-29T11:49:49.038 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.029+0000 7f7df4bbc640 1 -- 172.21.15.98:0/4145638323 wait complete. 2024-03-29T11:49:49.038 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.031+0000 7f7df4bbc640 10 monclient: build_initial_monmap 2024-03-29T11:49:49.038 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.031+0000 7f7df4bbc640 10 monclient: monmap: 2024-03-29T11:49:49.038 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:49:49.038 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:49:49.038 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:49:49.038 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:49:49.038 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:49:49.039 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:49.039 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:49:49.039 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:49:49.039 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:49:49.039 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:49.039 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.031+0000 7f7df4bbc640 1 librados: starting msgr at 2024-03-29T11:49:49.039 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.031+0000 7f7df4bbc640 1 librados: starting objecter 2024-03-29T11:49:49.039 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.031+0000 7f7df4bbc640 1 Processor -- start 2024-03-29T11:49:49.039 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.031+0000 7f7df4bbc640 1 -- start start 2024-03-29T11:49:49.039 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.031+0000 7f7df4bbc640 1 librados: setting wanted keys 2024-03-29T11:49:49.039 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.031+0000 7f7df4bbc640 1 librados: calling monclient init 2024-03-29T11:49:49.039 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.031+0000 7f7df4bbc640 10 monclient: init 2024-03-29T11:49:49.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.032+0000 7f7df4bbc640 10 monclient: _reopen_session rank -1 2024-03-29T11:49:49.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.032+0000 7f7df4bbc640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:49:49.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.032+0000 7f7df4bbc640 10 monclient(hunting): picked mon.noname-c con 0x7f7df00a9750 addr v1:172.21.15.98:6790/0 2024-03-29T11:49:49.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.032+0000 7f7df4bbc640 10 monclient(hunting): picked mon.noname-b con 0x7f7df0149ac0 addr v1:172.21.15.121:6789/0 2024-03-29T11:49:49.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.032+0000 7f7df4bbc640 10 monclient(hunting): picked mon.noname-a con 0x7f7df014fe50 addr v1:172.21.15.98:6789/0 2024-03-29T11:49:49.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.032+0000 7f7df4bbc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7df01db760 con 0x7f7df014fe50 2024-03-29T11:49:49.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.032+0000 7f7df4bbc640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7df00acb30 con 0x7f7df0149ac0 2024-03-29T11:49:49.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.032+0000 7f7df4bbc640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7df0144340 con 0x7f7df00a9750 2024-03-29T11:49:49.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.032+0000 7f7df4bbc640 10 monclient(hunting): _renew_subs 2024-03-29T11:49:49.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.032+0000 7f7df4bbc640 10 monclient(hunting): authenticate will time out at 896.117756s 2024-03-29T11:49:49.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.032+0000 7f7dee59c640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f7df00a9750 0x7f7df00ac420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:60882/0 (socket says 172.21.15.98:60882) 2024-03-29T11:49:49.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.032+0000 7f7dee59c640 1 -- 172.21.15.98:0/3671444670 learned_addr learned my addr 172.21.15.98:0/3671444670 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:49:49.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.032+0000 7f7ddeffd640 10 client.?.objecter ms_handle_connect 0x7f7df00a9750 2024-03-29T11:49:49.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.032+0000 7f7ddeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:49:49.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.032+0000 7f7ddeffd640 1 -- 172.21.15.98:0/3671444670 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1559311121 0 0) 0x7f7df0144340 con 0x7f7df00a9750 2024-03-29T11:49:49.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.032+0000 7f7ddeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:49:49.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.032+0000 7f7ddeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:49.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.032+0000 7f7ddeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:49.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.032+0000 7f7ddeffd640 10 monclient(hunting): my global_id is 4289 2024-03-29T11:49:49.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.033+0000 7f7ddeffd640 1 -- 172.21.15.98:0/3671444670 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dd0004eb0 con 0x7f7df00a9750 2024-03-29T11:49:49.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.033+0000 7f7ddeffd640 1 -- 172.21.15.98:0/3671444670 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2007857194 0 0) 0x7f7dd0004eb0 con 0x7f7df00a9750 2024-03-29T11:49:49.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.033+0000 7f7ddeffd640 1 -- 172.21.15.98:0/3671444670 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7df0144340 con 0x7f7df00a9750 2024-03-29T11:49:49.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.033+0000 7f7ddeffd640 1 -- 172.21.15.98:0/3671444670 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f7de0001910 con 0x7f7df00a9750 2024-03-29T11:49:49.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.033+0000 7f7ddeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:49.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.033+0000 7f7ddeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:49:49.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.033+0000 7f7ddeffd640 10 monclient(hunting): dump: 2024-03-29T11:49:49.044 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:49:49.044 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:49:49.044 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:49.044 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:49.044 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:49.044 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:49.044 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:49:49.044 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:49:49.045 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:49:49.045 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:49.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.033+0000 7f7ddeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:49:49.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.033+0000 7f7ddeffd640 1 -- 172.21.15.98:0/3671444670 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 870581698 0 0) 0x7f7df0144340 con 0x7f7df00a9750 2024-03-29T11:49:49.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.033+0000 7f7ddeffd640 1 -- 172.21.15.98:0/3671444670 >> v1:172.21.15.121:6789/0 conn(0x7f7df0149ac0 legacy=0x7f7df01dbc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:49.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.033+0000 7f7ddeffd640 1 -- 172.21.15.98:0/3671444670 >> v1:172.21.15.98:6789/0 conn(0x7f7df014fe50 legacy=0x7f7df013a2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:49:49.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.036+0000 7f7ddeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:49:49.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.036+0000 7f7ddeffd640 1 monclient: found mon.c 2024-03-29T11:49:49.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.036+0000 7f7ddeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:49:49.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.036+0000 7f7ddeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:49:49.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.036+0000 7f7ddeffd640 1 -- 172.21.15.98:0/3671444670 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df01478c0 con 0x7f7df00a9750 2024-03-29T11:49:49.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.036+0000 7f7ddeffd640 10 monclient: _finish_auth 0 2024-03-29T11:49:49.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.036+0000 7f7ddeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:49:49.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.036+0000 7f7ddeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:49:49.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.036+0000 7f7df4bbc640 5 monclient: authenticate success, global_id 4289 2024-03-29T11:49:49.046 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.036+0000 7f7df4bbc640 10 monclient: _renew_subs 2024-03-29T11:49:49.046 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.036+0000 7f7df4bbc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:49:49.046 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.036+0000 7f7df4bbc640 1 -- 172.21.15.98:0/3671444670 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7df00abcd0 con 0x7f7df00a9750 2024-03-29T11:49:49.046 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.036+0000 7f7df4bbc640 10 client.4289.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:49:49.046 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.036+0000 7f7df4bbc640 10 monclient: _renew_subs 2024-03-29T11:49:49.046 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.036+0000 7f7df4bbc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:49:49.046 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.036+0000 7f7df4bbc640 1 -- 172.21.15.98:0/3671444670 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7df01dc570 con 0x7f7df00a9750 2024-03-29T11:49:49.046 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.036+0000 7f7df4bbc640 1 librados: init done 2024-03-29T11:49:49.047 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.036+0000 7f7ddeffd640 1 -- 172.21.15.98:0/3671444670 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7de0001cb0 con 0x7f7df00a9750 2024-03-29T11:49:49.047 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.036+0000 7f7ddeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:49:49.048 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.036+0000 7f7ddeffd640 1 -- 172.21.15.98:0/3671444670 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f7de0004c30 con 0x7f7df00a9750 2024-03-29T11:49:49.048 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.036+0000 7f7ddeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:49.048 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.036+0000 7f7ddeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:49:49.048 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.036+0000 7f7ddeffd640 10 monclient: dump: 2024-03-29T11:49:49.048 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:49:49.048 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:49:49.048 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:49.048 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:49.048 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:49.048 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:49.048 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:49:49.048 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:49:49.048 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:49:49.049 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:49.049 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.036+0000 7f7decd99640 4 set_mon_vals no callback set 2024-03-29T11:49:49.049 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.037+0000 7f7df4bbc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:49.049 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.037+0000 7f7df4bbc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:49.049 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.037+0000 7f7df4bbc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:49:49.049 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.037+0000 7f7df4bbc640 1 -- 172.21.15.98:0/3671444670 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dd4005080 con 0x7f7df00a9750 2024-03-29T11:49:49.049 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.038+0000 7f7ddeffd640 1 -- 172.21.15.98:0/3671444670 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f7de001ac00 con 0x7f7df00a9750 2024-03-29T11:49:49.049 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.039+0000 7f7ddeffd640 1 -- 172.21.15.98:0/3671444670 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f7de0084c50 con 0x7f7df00a9750 2024-03-29T11:49:49.049 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.039+0000 7f7ddeffd640 10 client.4289.objecter ms_dispatch 0x7f7df0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:49:49.049 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.039+0000 7f7ddeffd640 3 client.4289.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:49:49.049 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.039+0000 7f7ddeffd640 3 client.4289.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:49:49.049 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.039+0000 7f7ddeffd640 20 client.4289.objecter dump_active .. 0 homeless 2024-03-29T11:49:49.049 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.039+0000 7f7dedd9b640 10 monclient: get_auth_request con 0x7f7dd006db10 auth_method 0 2024-03-29T11:49:49.049 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.040+0000 7f7ddeffd640 10 client.4289.objecter ms_handle_connect 0x7f7dd006db10 2024-03-29T11:49:49.054 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.048+0000 7f7ddeffd640 1 -- 172.21.15.98:0/3671444670 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7de0050360 con 0x7f7df00a9750 2024-03-29T11:49:49.054 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.048+0000 7f7ddeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:49.054 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.048+0000 7f7ddeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:49:49.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.358+0000 7f7df4bbc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:49.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.358+0000 7f7df4bbc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:49.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.358+0000 7f7df4bbc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:49:49.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.358+0000 7f7df4bbc640 1 -- 172.21.15.98:0/3671444670 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7dd4005330 con 0x7f7df00a9750 2024-03-29T11:49:49.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.359+0000 7f7ddeffd640 1 -- 172.21.15.98:0/3671444670 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f7de0052460 con 0x7f7df00a9750 2024-03-29T11:49:49.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.359+0000 7f7ddeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:49.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.359+0000 7f7ddeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:49:49.366 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:49:49.366 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:49:49.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.363+0000 7f7df4bbc640 10 librados: watch_flush enter 2024-03-29T11:49:49.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.363+0000 7f7df4bbc640 10 librados: watch_flush exit 2024-03-29T11:49:49.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.363+0000 7f7df4bbc640 20 client.4289.objecter shutdown clearing up homeless session... 2024-03-29T11:49:49.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.363+0000 7f7df4bbc640 10 client.4289.objecter successfully canceled tick 2024-03-29T11:49:49.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.363+0000 7f7df4bbc640 1 -- 172.21.15.98:0/3671444670 >> v1:172.21.15.98:6816/4065186974 conn(0x7f7dd006db10 legacy=0x7f7dd006ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:49.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.363+0000 7f7df4bbc640 10 monclient: shutdown 2024-03-29T11:49:49.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.363+0000 7f7df4bbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:49:49.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.363+0000 7f7df4bbc640 1 -- 172.21.15.98:0/3671444670 >> v1:172.21.15.98:6790/0 conn(0x7f7df00a9750 legacy=0x7f7df00ac420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:49.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.363+0000 7f7df4bbc640 1 -- 172.21.15.98:0/3671444670 shutdown_connections 2024-03-29T11:49:49.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.363+0000 7f7df4bbc640 1 -- 172.21.15.98:0/3671444670 >> 172.21.15.98:0/3671444670 conn(0x7f7df00b35e0 msgr2=0x7f7df0141560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:49:49.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.363+0000 7f7df4bbc640 1 -- 172.21.15.98:0/3671444670 shutdown_connections 2024-03-29T11:49:49.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.363+0000 7f7df4bbc640 1 -- 172.21.15.98:0/3671444670 wait complete. 2024-03-29T11:49:49.371 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:49.363+0000 7f7df4bbc640 1 librados: shutdown 2024-03-29T11:49:49.384 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:49:56.191 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:49:56.338 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.332+0000 7feea876b640 10 monclient: get_monmap_and_config 2024-03-29T11:49:56.338 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.332+0000 7feea876b640 10 monclient: build_initial_monmap 2024-03-29T11:49:56.338 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.332+0000 7feea876b640 10 monclient: monmap: 2024-03-29T11:49:56.338 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:49:56.339 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:49:56.339 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:49:56.333051+0000 2024-03-29T11:49:56.339 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:49:56.333051+0000 2024-03-29T11:49:56.339 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:49:56.339 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:56.339 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:49:56.339 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:49:56.339 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:49:56.339 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:56.339 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.333+0000 7feea876b640 1 Processor -- start 2024-03-29T11:49:56.341 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.333+0000 7feea876b640 1 -- start start 2024-03-29T11:49:56.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.333+0000 7feea876b640 10 monclient: init 2024-03-29T11:49:56.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.334+0000 7feea876b640 10 monclient: _reopen_session rank -1 2024-03-29T11:49:56.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.334+0000 7feea876b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:49:56.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.334+0000 7feea876b640 10 monclient(hunting): picked mon.noname-b con 0x7feea01422b0 addr v1:172.21.15.98:6790/0 2024-03-29T11:49:56.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.334+0000 7feea876b640 10 monclient(hunting): picked mon.noname-c con 0x7feea0142690 addr v1:172.21.15.121:6789/0 2024-03-29T11:49:56.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.334+0000 7feea876b640 10 monclient(hunting): picked mon.noname-a con 0x7feea00b17d0 addr v1:172.21.15.98:6789/0 2024-03-29T11:49:56.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.334+0000 7feea876b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feea0142c80 con 0x7feea00b17d0 2024-03-29T11:49:56.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.334+0000 7feea876b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feea00aae40 con 0x7feea0142690 2024-03-29T11:49:56.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.334+0000 7feea876b640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feea013fbd0 con 0x7feea01422b0 2024-03-29T11:49:56.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.334+0000 7feea876b640 10 monclient(hunting): _renew_subs 2024-03-29T11:49:56.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.334+0000 7feea876b640 10 monclient(hunting): authenticate will time out at 903.420045s 2024-03-29T11:49:56.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.336+0000 7feea5d06640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7feea0142690 0x7feea00b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:49078/0 (socket says 172.21.15.98:49078) 2024-03-29T11:49:56.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.336+0000 7feea5d06640 1 -- 172.21.15.98:0/348009218 learned_addr learned my addr 172.21.15.98:0/348009218 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:49:56.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.337+0000 7feea5505640 1 -- 172.21.15.98:0/348009218 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2470606428 0 0) 0x7feea00aae40 con 0x7feea0142690 2024-03-29T11:49:56.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.337+0000 7feea5505640 10 monclient(hunting): _negotiate 2024-03-29T11:49:56.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.337+0000 7feea5505640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:56.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.337+0000 7feea5505640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:56.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.337+0000 7feea5505640 10 monclient(hunting): my global_id is 4306 2024-03-29T11:49:56.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.337+0000 7feea5505640 1 -- 172.21.15.98:0/348009218 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee88003520 con 0x7feea0142690 2024-03-29T11:49:56.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.337+0000 7feea5505640 1 -- 172.21.15.98:0/348009218 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1357079597 0 0) 0x7fee88003520 con 0x7feea0142690 2024-03-29T11:49:56.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.337+0000 7feea5505640 1 -- 172.21.15.98:0/348009218 >> v1:172.21.15.98:6790/0 conn(0x7feea01422b0 legacy=0x7feea013f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:49:56.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.338+0000 7feea5505640 1 -- 172.21.15.98:0/348009218 >> v1:172.21.15.98:6789/0 conn(0x7feea00b17d0 legacy=0x7feea00b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:49:56.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.338+0000 7feea5505640 10 monclient: _finish_hunting 0 2024-03-29T11:49:56.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.338+0000 7feea5505640 1 monclient: found mon.noname-c 2024-03-29T11:49:56.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.338+0000 7feea5505640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:49:56.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.338+0000 7feea5505640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:49:56.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.338+0000 7feea5505640 1 -- 172.21.15.98:0/348009218 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea0151940 con 0x7feea0142690 2024-03-29T11:49:56.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.338+0000 7feea5505640 10 monclient: _finish_auth 0 2024-03-29T11:49:56.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.338+0000 7feea5505640 10 monclient: _check_auth_tickets 2024-03-29T11:49:56.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.338+0000 7feea5505640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:49:56.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.338+0000 7feea5505640 1 -- 172.21.15.98:0/348009218 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fee94001690 con 0x7feea0142690 2024-03-29T11:49:56.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.338+0000 7feea5505640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:56.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.338+0000 7feea5505640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:49:56.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.338+0000 7feea5505640 10 monclient: dump: 2024-03-29T11:49:56.345 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:49:56.345 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:49:56.345 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:56.345 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:56.345 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:56.345 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:56.346 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:49:56.346 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:49:56.346 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:49:56.346 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:56.346 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.338+0000 7feea876b640 5 monclient: authenticate success, global_id 4306 2024-03-29T11:49:56.346 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.338+0000 7feea876b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:49:56.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.339+0000 7feea5505640 1 -- 172.21.15.98:0/348009218 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee940017f0 con 0x7feea0142690 2024-03-29T11:49:56.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.339+0000 7feea5505640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:49:56.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.339+0000 7feea5505640 1 -- 172.21.15.98:0/348009218 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fee94001c40 con 0x7feea0142690 2024-03-29T11:49:56.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.339+0000 7feea5505640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:56.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.339+0000 7feea5505640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:49:56.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.339+0000 7feea5505640 10 monclient: dump: 2024-03-29T11:49:56.347 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:49:56.347 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:49:56.347 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:56.347 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:56.347 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:56.347 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:56.347 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:49:56.348 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:49:56.348 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:49:56.348 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:56.348 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.339+0000 7feea876b640 10 monclient: get_monmap_and_config success 2024-03-29T11:49:56.348 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.339+0000 7feea876b640 4 set_mon_vals no callback set 2024-03-29T11:49:56.348 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.340+0000 7feea876b640 10 monclient: shutdown 2024-03-29T11:49:56.348 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.340+0000 7feea876b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:49:56.348 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.340+0000 7feea876b640 1 -- 172.21.15.98:0/348009218 >> v1:172.21.15.121:6789/0 conn(0x7feea0142690 legacy=0x7feea00b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:56.349 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.343+0000 7feea876b640 1 -- 172.21.15.98:0/348009218 shutdown_connections 2024-03-29T11:49:56.349 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.343+0000 7feea876b640 1 -- 172.21.15.98:0/348009218 >> 172.21.15.98:0/348009218 conn(0x7feea0066a00 msgr2=0x7feea0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:49:56.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.343+0000 7feea876b640 1 -- 172.21.15.98:0/348009218 shutdown_connections 2024-03-29T11:49:56.352 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.344+0000 7feea876b640 1 -- 172.21.15.98:0/348009218 wait complete. 2024-03-29T11:49:56.352 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.344+0000 7feea876b640 10 monclient: build_initial_monmap 2024-03-29T11:49:56.352 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.344+0000 7feea876b640 10 monclient: monmap: 2024-03-29T11:49:56.352 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:49:56.352 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:49:56.352 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:49:56.352 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:49:56.352 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:49:56.352 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:56.352 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:49:56.352 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:49:56.353 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:49:56.353 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:56.353 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.345+0000 7feea876b640 1 librados: starting msgr at 2024-03-29T11:49:56.353 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.345+0000 7feea876b640 1 librados: starting objecter 2024-03-29T11:49:56.353 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.345+0000 7feea876b640 1 Processor -- start 2024-03-29T11:49:56.354 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.345+0000 7feea876b640 1 -- start start 2024-03-29T11:49:56.354 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.345+0000 7feea876b640 1 librados: setting wanted keys 2024-03-29T11:49:56.354 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.345+0000 7feea876b640 1 librados: calling monclient init 2024-03-29T11:49:56.355 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.345+0000 7feea876b640 10 monclient: init 2024-03-29T11:49:56.355 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.346+0000 7feea876b640 10 monclient: _reopen_session rank -1 2024-03-29T11:49:56.355 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.346+0000 7feea876b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:49:56.355 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.346+0000 7feea876b640 10 monclient(hunting): picked mon.noname-b con 0x7feea00b17d0 addr v1:172.21.15.121:6789/0 2024-03-29T11:49:56.355 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.346+0000 7feea876b640 10 monclient(hunting): picked mon.noname-a con 0x7feea01422b0 addr v1:172.21.15.98:6789/0 2024-03-29T11:49:56.355 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.346+0000 7feea876b640 10 monclient(hunting): picked mon.noname-c con 0x7feea0142690 addr v1:172.21.15.98:6790/0 2024-03-29T11:49:56.355 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.346+0000 7feea876b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feea006ffc0 con 0x7feea01422b0 2024-03-29T11:49:56.355 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.346+0000 7feea876b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feea0130550 con 0x7feea00b17d0 2024-03-29T11:49:56.355 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.346+0000 7feea876b640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feea0071f40 con 0x7feea0142690 2024-03-29T11:49:56.355 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.346+0000 7feea876b640 10 monclient(hunting): _renew_subs 2024-03-29T11:49:56.355 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.346+0000 7feea876b640 10 monclient(hunting): authenticate will time out at 903.431474s 2024-03-29T11:49:56.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.350+0000 7feea6507640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7feea00b17d0 0x7feea0070750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:49354/0 (socket says 172.21.15.98:49354) 2024-03-29T11:49:56.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.350+0000 7feea6507640 1 -- 172.21.15.98:0/3158652885 learned_addr learned my addr 172.21.15.98:0/3158652885 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:49:56.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.352+0000 7fee9effd640 10 client.?.objecter ms_handle_connect 0x7feea00b17d0 2024-03-29T11:49:56.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.352+0000 7fee9effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:49:56.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.352+0000 7fee9effd640 1 -- 172.21.15.98:0/3158652885 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2673878569 0 0) 0x7feea0130550 con 0x7feea00b17d0 2024-03-29T11:49:56.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.352+0000 7fee9effd640 10 monclient(hunting): _negotiate 2024-03-29T11:49:56.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.352+0000 7fee9effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:56.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.352+0000 7fee9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:56.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.352+0000 7fee9effd640 10 monclient(hunting): my global_id is 4309 2024-03-29T11:49:56.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.352+0000 7fee9effd640 1 -- 172.21.15.98:0/3158652885 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee8c004eb0 con 0x7feea00b17d0 2024-03-29T11:49:56.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.352+0000 7fee9effd640 10 client.?.objecter ms_handle_connect 0x7feea01422b0 2024-03-29T11:49:56.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.352+0000 7fee9effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:49:56.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.352+0000 7fee9effd640 10 client.?.objecter ms_handle_connect 0x7feea0142690 2024-03-29T11:49:56.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.352+0000 7fee9effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:49:56.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.352+0000 7fee9effd640 1 -- 172.21.15.98:0/3158652885 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2767706574 0 0) 0x7feea006ffc0 con 0x7feea01422b0 2024-03-29T11:49:56.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.352+0000 7fee9effd640 10 monclient(hunting): _negotiate 2024-03-29T11:49:56.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.352+0000 7fee9effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:56.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.352+0000 7fee9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:56.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.352+0000 7fee9effd640 10 monclient(hunting): my global_id is 4281 2024-03-29T11:49:56.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.352+0000 7fee9effd640 1 -- 172.21.15.98:0/3158652885 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feea0130550 con 0x7feea01422b0 2024-03-29T11:49:56.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.352+0000 7fee9effd640 1 -- 172.21.15.98:0/3158652885 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2733468450 0 0) 0x7fee8c004eb0 con 0x7feea00b17d0 2024-03-29T11:49:56.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.352+0000 7fee9effd640 1 -- 172.21.15.98:0/3158652885 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feea006ffc0 con 0x7feea00b17d0 2024-03-29T11:49:56.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.352+0000 7fee9effd640 1 -- 172.21.15.98:0/3158652885 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 892525874 0 0) 0x7feea0071f40 con 0x7feea0142690 2024-03-29T11:49:56.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.352+0000 7fee9effd640 10 monclient(hunting): _negotiate 2024-03-29T11:49:56.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.352+0000 7fee9effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:49:56.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.352+0000 7fee9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:49:56.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.352+0000 7fee9effd640 10 monclient(hunting): my global_id is 4292 2024-03-29T11:49:56.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.352+0000 7fee9effd640 1 -- 172.21.15.98:0/3158652885 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee8c004eb0 con 0x7feea0142690 2024-03-29T11:49:56.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.353+0000 7fee9effd640 1 -- 172.21.15.98:0/3158652885 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fee90004400 con 0x7feea00b17d0 2024-03-29T11:49:56.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.353+0000 7fee9effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:56.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.353+0000 7fee9effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:49:56.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.353+0000 7fee9effd640 10 monclient(hunting): dump: 2024-03-29T11:49:56.365 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:49:56.365 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:49:56.365 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:56.365 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:56.365 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:56.365 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:56.365 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:49:56.365 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:49:56.365 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:49:56.365 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:56.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.353+0000 7fee9effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:49:56.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.353+0000 7fee9effd640 1 -- 172.21.15.98:0/3158652885 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3469045545 0 0) 0x7fee8c004eb0 con 0x7feea0142690 2024-03-29T11:49:56.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.353+0000 7fee9effd640 1 -- 172.21.15.98:0/3158652885 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feea0071f40 con 0x7feea0142690 2024-03-29T11:49:56.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.353+0000 7fee9effd640 1 -- 172.21.15.98:0/3158652885 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 49022778 0 0) 0x7feea0130550 con 0x7feea01422b0 2024-03-29T11:49:56.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.353+0000 7fee9effd640 1 -- 172.21.15.98:0/3158652885 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee8c004eb0 con 0x7feea01422b0 2024-03-29T11:49:56.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.353+0000 7fee9effd640 1 -- 172.21.15.98:0/3158652885 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fee98001490 con 0x7feea0142690 2024-03-29T11:49:56.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.353+0000 7fee9effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:56.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.353+0000 7fee9effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:49:56.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.353+0000 7fee9effd640 10 monclient(hunting): dump: 2024-03-29T11:49:56.366 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:49:56.366 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:49:56.366 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:56.366 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:56.366 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:56.366 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:56.366 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:49:56.367 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:49:56.367 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:49:56.367 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:56.367 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.353+0000 7fee9effd640 1 -- 172.21.15.98:0/3158652885 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fee94001b00 con 0x7feea01422b0 2024-03-29T11:49:56.367 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.353+0000 7fee9effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:56.367 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.353+0000 7fee9effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:49:56.367 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.353+0000 7fee9effd640 10 monclient(hunting): dump: 2024-03-29T11:49:56.367 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:49:56.367 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:49:56.367 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:56.367 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:56.367 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:56.367 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:56.367 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:49:56.368 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:49:56.368 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:49:56.368 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:56.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.353+0000 7fee9effd640 1 -- 172.21.15.98:0/3158652885 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3654760331 0 0) 0x7feea0071f40 con 0x7feea0142690 2024-03-29T11:49:56.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.353+0000 7fee9effd640 1 -- 172.21.15.98:0/3158652885 >> v1:172.21.15.121:6789/0 conn(0x7feea00b17d0 legacy=0x7feea0070750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:56.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.353+0000 7fee9effd640 1 -- 172.21.15.98:0/3158652885 >> v1:172.21.15.98:6789/0 conn(0x7feea01422b0 legacy=0x7feea012fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:56.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.353+0000 7fee9effd640 10 monclient: _finish_hunting 0 2024-03-29T11:49:56.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.353+0000 7fee9effd640 1 monclient: found mon.c 2024-03-29T11:49:56.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.353+0000 7fee9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:49:56.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.353+0000 7fee9effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:49:56.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.353+0000 7fee9effd640 1 -- 172.21.15.98:0/3158652885 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea00720d0 con 0x7feea0142690 2024-03-29T11:49:56.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.353+0000 7fee9effd640 10 monclient: _finish_auth 0 2024-03-29T11:49:56.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.353+0000 7fee9effd640 10 monclient: _check_auth_tickets 2024-03-29T11:49:56.369 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.353+0000 7fee9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:49:56.369 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.353+0000 7feea876b640 5 monclient: authenticate success, global_id 4292 2024-03-29T11:49:56.369 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.353+0000 7feea876b640 10 monclient: _renew_subs 2024-03-29T11:49:56.369 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.353+0000 7feea876b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:49:56.369 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.353+0000 7feea876b640 1 -- 172.21.15.98:0/3158652885 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feea00711b0 con 0x7feea0142690 2024-03-29T11:49:56.369 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.353+0000 7feea876b640 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:49:56.369 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.353+0000 7feea876b640 10 monclient: _renew_subs 2024-03-29T11:49:56.369 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.353+0000 7feea876b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:49:56.369 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.353+0000 7feea876b640 1 -- 172.21.15.98:0/3158652885 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feea0132920 con 0x7feea0142690 2024-03-29T11:49:56.369 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.353+0000 7feea876b640 1 librados: init done 2024-03-29T11:49:56.369 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.354+0000 7fee9effd640 1 -- 172.21.15.98:0/3158652885 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee980019a0 con 0x7feea0142690 2024-03-29T11:49:56.369 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.354+0000 7fee9effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:49:56.369 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.355+0000 7feea4d04640 4 set_mon_vals no callback set 2024-03-29T11:49:56.369 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.355+0000 7feea876b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:56.369 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.356+0000 7fee9effd640 1 -- 172.21.15.98:0/3158652885 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fee980037d0 con 0x7feea0142690 2024-03-29T11:49:56.369 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.356+0000 7feea876b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:56.369 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.356+0000 7feea876b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:49:56.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.356+0000 7feea876b640 1 -- 172.21.15.98:0/3158652885 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feea0070ed0 con 0x7feea0142690 2024-03-29T11:49:56.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.356+0000 7fee9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:49:56.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.356+0000 7fee9effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:49:56.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.356+0000 7fee9effd640 10 monclient: dump: 2024-03-29T11:49:56.370 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:49:56.370 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:49:56.370 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:56.370 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:49:56.370 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:49:56.370 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:49:56.370 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:49:56.370 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:49:56.370 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:49:56.370 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:49:56.371 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.357+0000 7fee9effd640 1 -- 172.21.15.98:0/3158652885 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7fee980197a0 con 0x7feea0142690 2024-03-29T11:49:56.371 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.357+0000 7fee9effd640 1 -- 172.21.15.98:0/3158652885 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7fee98083480 con 0x7feea0142690 2024-03-29T11:49:56.371 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.357+0000 7fee9effd640 10 client.4292.objecter ms_dispatch 0x7feea0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:49:56.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.357+0000 7fee9effd640 3 client.4292.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:49:56.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.357+0000 7fee9effd640 3 client.4292.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:49:56.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.357+0000 7feea6507640 10 monclient: get_auth_request con 0x7fee8c06e2b0 auth_method 0 2024-03-29T11:49:56.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.357+0000 7fee9effd640 20 client.4292.objecter dump_active .. 0 homeless 2024-03-29T11:49:56.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.358+0000 7fee9effd640 10 client.4292.objecter ms_handle_connect 0x7fee8c06e2b0 2024-03-29T11:49:56.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.366+0000 7fee9effd640 1 -- 172.21.15.98:0/3158652885 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feea0070ed0 con 0x7feea0142690 2024-03-29T11:49:56.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.366+0000 7fee9effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:49:56.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.366+0000 7fee9effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:49:56.697 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.691+0000 7feea876b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:56.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.691+0000 7feea876b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:56.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.691+0000 7feea876b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:49:56.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.691+0000 7feea876b640 1 -- 172.21.15.98:0/3158652885 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feea006fdc0 con 0x7feea0142690 2024-03-29T11:49:56.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.692+0000 7fee9effd640 1 -- 172.21.15.98:0/3158652885 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7feea0070ed0 con 0x7feea0142690 2024-03-29T11:49:56.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.692+0000 7fee9effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:49:56.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.692+0000 7fee9effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:49:56.699 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:49:56.699 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:49:56.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.695+0000 7feea876b640 10 librados: watch_flush enter 2024-03-29T11:49:56.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.696+0000 7feea876b640 10 librados: watch_flush exit 2024-03-29T11:49:56.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.696+0000 7feea876b640 20 client.4292.objecter shutdown clearing up homeless session... 2024-03-29T11:49:56.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.696+0000 7feea876b640 10 client.4292.objecter successfully canceled tick 2024-03-29T11:49:56.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.696+0000 7feea876b640 1 -- 172.21.15.98:0/3158652885 >> v1:172.21.15.98:6816/4065186974 conn(0x7fee8c06e2b0 legacy=0x7fee8c070770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:56.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.696+0000 7feea876b640 10 monclient: shutdown 2024-03-29T11:49:56.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.696+0000 7feea876b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:49:56.703 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.696+0000 7feea876b640 1 -- 172.21.15.98:0/3158652885 >> v1:172.21.15.98:6790/0 conn(0x7feea0142690 legacy=0x7feea01ee830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:49:56.703 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.696+0000 7feea876b640 1 -- 172.21.15.98:0/3158652885 shutdown_connections 2024-03-29T11:49:56.703 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.696+0000 7feea876b640 1 -- 172.21.15.98:0/3158652885 >> 172.21.15.98:0/3158652885 conn(0x7feea0066a00 msgr2=0x7feea00b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:49:56.703 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.697+0000 7feea876b640 1 -- 172.21.15.98:0/3158652885 shutdown_connections 2024-03-29T11:49:56.703 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.697+0000 7feea876b640 1 -- 172.21.15.98:0/3158652885 wait complete. 2024-03-29T11:49:56.703 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:49:56.697+0000 7feea876b640 1 librados: shutdown 2024-03-29T11:49:56.720 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:50:03.525 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:50:03.682 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.675+0000 7f9a4f796640 10 monclient: get_monmap_and_config 2024-03-29T11:50:03.683 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.675+0000 7f9a4f796640 10 monclient: build_initial_monmap 2024-03-29T11:50:03.683 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.675+0000 7f9a4f796640 10 monclient: monmap: 2024-03-29T11:50:03.683 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:50:03.683 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:50:03.683 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:50:03.676329+0000 2024-03-29T11:50:03.683 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:50:03.676329+0000 2024-03-29T11:50:03.683 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:50:03.683 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:03.683 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:50:03.683 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:50:03.683 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:50:03.683 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:03.683 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.676+0000 7f9a4f796640 1 Processor -- start 2024-03-29T11:50:03.683 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.676+0000 7f9a4f796640 1 -- start start 2024-03-29T11:50:03.683 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.676+0000 7f9a4f796640 10 monclient: init 2024-03-29T11:50:03.687 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.677+0000 7f9a4f796640 10 monclient: _reopen_session rank -1 2024-03-29T11:50:03.687 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.677+0000 7f9a4f796640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:50:03.687 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.677+0000 7f9a4f796640 10 monclient(hunting): picked mon.noname-a con 0x7f9a480a9930 addr v1:172.21.15.98:6789/0 2024-03-29T11:50:03.687 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.677+0000 7f9a4f796640 10 monclient(hunting): picked mon.noname-b con 0x7f9a48149dc0 addr v1:172.21.15.98:6790/0 2024-03-29T11:50:03.687 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.680+0000 7f9a4cd31640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f9a48149dc0 0x7f9a4814a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:50391/0 (socket says 172.21.15.98:50391) 2024-03-29T11:50:03.690 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.681+0000 7f9a4d532640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f9a480a9930 0x7f9a48143a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51533/0 (socket says 172.21.15.98:51533) 2024-03-29T11:50:03.690 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.683+0000 7f9a4f796640 10 monclient(hunting): picked mon.noname-c con 0x7f9a4814ff40 addr v1:172.21.15.121:6789/0 2024-03-29T11:50:03.690 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.683+0000 7f9a4f796640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a480aae60 con 0x7f9a480a9930 2024-03-29T11:50:03.690 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.683+0000 7f9a4f796640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a480b4310 con 0x7f9a4814ff40 2024-03-29T11:50:03.690 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.683+0000 7f9a4f796640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a480afdd0 con 0x7f9a48149dc0 2024-03-29T11:50:03.690 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.683+0000 7f9a4f796640 10 monclient(hunting): _renew_subs 2024-03-29T11:50:03.690 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.683+0000 7f9a4f796640 10 monclient(hunting): authenticate will time out at 910.768864s 2024-03-29T11:50:03.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.684+0000 7f9a4cd31640 1 -- 172.21.15.98:0/2795422588 learned_addr learned my addr 172.21.15.98:0/2795422588 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:50:03.692 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.685+0000 7f9a3ffff640 1 -- 172.21.15.98:0/2795422588 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 217325093 0 0) 0x7f9a480afdd0 con 0x7f9a48149dc0 2024-03-29T11:50:03.692 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.685+0000 7f9a3ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:50:03.692 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.686+0000 7f9a3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:03.692 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.686+0000 7f9a3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:03.692 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.686+0000 7f9a3ffff640 10 monclient(hunting): my global_id is 4295 2024-03-29T11:50:03.692 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.686+0000 7f9a3ffff640 1 -- 172.21.15.98:0/2795422588 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a34003520 con 0x7f9a48149dc0 2024-03-29T11:50:03.692 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.686+0000 7f9a3ffff640 1 -- 172.21.15.98:0/2795422588 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1097153815 0 0) 0x7f9a480aae60 con 0x7f9a480a9930 2024-03-29T11:50:03.692 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.686+0000 7f9a3ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:50:03.692 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.686+0000 7f9a3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:03.692 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.686+0000 7f9a3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:03.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.686+0000 7f9a3ffff640 10 monclient(hunting): my global_id is 4284 2024-03-29T11:50:03.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.686+0000 7f9a3ffff640 1 -- 172.21.15.98:0/2795422588 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a480afdd0 con 0x7f9a480a9930 2024-03-29T11:50:03.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.686+0000 7f9a3ffff640 1 -- 172.21.15.98:0/2795422588 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1727302844 0 0) 0x7f9a34003520 con 0x7f9a48149dc0 2024-03-29T11:50:03.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.686+0000 7f9a3ffff640 1 -- 172.21.15.98:0/2795422588 >> v1:172.21.15.121:6789/0 conn(0x7f9a4814ff40 legacy=0x7f9a48152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:50:03.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.686+0000 7f9a3ffff640 1 -- 172.21.15.98:0/2795422588 >> v1:172.21.15.98:6789/0 conn(0x7f9a480a9930 legacy=0x7f9a48143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:03.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.686+0000 7f9a3ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:50:03.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.686+0000 7f9a3ffff640 1 monclient: found mon.noname-b 2024-03-29T11:50:03.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.686+0000 7f9a3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:50:03.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.686+0000 7f9a3ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:50:03.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.686+0000 7f9a3ffff640 1 -- 172.21.15.98:0/2795422588 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a4814a950 con 0x7f9a48149dc0 2024-03-29T11:50:03.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.686+0000 7f9a3ffff640 10 monclient: _finish_auth 0 2024-03-29T11:50:03.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.686+0000 7f9a3ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:50:03.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.686+0000 7f9a3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:50:03.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.686+0000 7f9a3ffff640 1 -- 172.21.15.98:0/2795422588 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f9a38001690 con 0x7f9a48149dc0 2024-03-29T11:50:03.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.686+0000 7f9a3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:03.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.686+0000 7f9a3ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:50:03.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.686+0000 7f9a3ffff640 10 monclient: dump: 2024-03-29T11:50:03.694 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:50:03.694 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:50:03.694 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:03.694 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:03.694 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:03.694 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:03.694 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:50:03.694 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:50:03.695 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:50:03.695 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:03.695 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.686+0000 7f9a3ffff640 1 -- 172.21.15.98:0/2795422588 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a380017f0 con 0x7f9a48149dc0 2024-03-29T11:50:03.695 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.686+0000 7f9a3ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:50:03.695 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.686+0000 7f9a3ffff640 1 -- 172.21.15.98:0/2795422588 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f9a38001c40 con 0x7f9a48149dc0 2024-03-29T11:50:03.695 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.686+0000 7f9a3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:03.695 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.686+0000 7f9a3ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:50:03.695 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.686+0000 7f9a3ffff640 10 monclient: dump: 2024-03-29T11:50:03.695 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:50:03.695 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:50:03.695 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:03.695 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:03.695 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:03.695 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:03.695 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:50:03.695 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:50:03.695 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:50:03.696 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:03.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.688+0000 7f9a4f796640 5 monclient: authenticate success, global_id 4295 2024-03-29T11:50:03.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.688+0000 7f9a4f796640 10 monclient: get_monmap_and_config success 2024-03-29T11:50:03.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.688+0000 7f9a4f796640 4 set_mon_vals no callback set 2024-03-29T11:50:03.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.689+0000 7f9a4f796640 10 monclient: shutdown 2024-03-29T11:50:03.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.689+0000 7f9a4f796640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:50:03.697 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.689+0000 7f9a4f796640 1 -- 172.21.15.98:0/2795422588 >> v1:172.21.15.98:6790/0 conn(0x7f9a48149dc0 legacy=0x7f9a4814a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:03.697 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.689+0000 7f9a4f796640 1 -- 172.21.15.98:0/2795422588 shutdown_connections 2024-03-29T11:50:03.697 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.689+0000 7f9a4f796640 1 -- 172.21.15.98:0/2795422588 >> 172.21.15.98:0/2795422588 conn(0x7f9a480b35e0 msgr2=0x7f9a480b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:50:03.697 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.690+0000 7f9a4f796640 1 -- 172.21.15.98:0/2795422588 shutdown_connections 2024-03-29T11:50:03.697 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.690+0000 7f9a4f796640 1 -- 172.21.15.98:0/2795422588 wait complete. 2024-03-29T11:50:03.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.691+0000 7f9a4f796640 10 monclient: build_initial_monmap 2024-03-29T11:50:03.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.691+0000 7f9a4f796640 10 monclient: monmap: 2024-03-29T11:50:03.698 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:50:03.698 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:50:03.698 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:50:03.698 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:50:03.698 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:50:03.698 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:03.698 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:50:03.698 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:50:03.698 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:50:03.698 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:03.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.692+0000 7f9a4f796640 1 librados: starting msgr at 2024-03-29T11:50:03.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.692+0000 7f9a4f796640 1 librados: starting objecter 2024-03-29T11:50:03.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.692+0000 7f9a4f796640 1 Processor -- start 2024-03-29T11:50:03.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.692+0000 7f9a4f796640 1 -- start start 2024-03-29T11:50:03.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.692+0000 7f9a4f796640 1 librados: setting wanted keys 2024-03-29T11:50:03.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.692+0000 7f9a4f796640 1 librados: calling monclient init 2024-03-29T11:50:03.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.692+0000 7f9a4f796640 10 monclient: init 2024-03-29T11:50:03.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.693+0000 7f9a4f796640 10 monclient: _reopen_session rank -1 2024-03-29T11:50:03.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.693+0000 7f9a4f796640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:50:03.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.693+0000 7f9a4f796640 10 monclient(hunting): picked mon.noname-c con 0x7f9a480a9930 addr v1:172.21.15.98:6790/0 2024-03-29T11:50:03.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.693+0000 7f9a4f796640 10 monclient(hunting): picked mon.noname-b con 0x7f9a48149dc0 addr v1:172.21.15.121:6789/0 2024-03-29T11:50:03.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.696+0000 7f9a4f796640 10 monclient(hunting): picked mon.noname-a con 0x7f9a4814ff40 addr v1:172.21.15.98:6789/0 2024-03-29T11:50:03.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.696+0000 7f9a4f796640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a48072620 con 0x7f9a4814ff40 2024-03-29T11:50:03.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.696+0000 7f9a4f796640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a480719f0 con 0x7f9a48149dc0 2024-03-29T11:50:03.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.696+0000 7f9a4f796640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a480715f0 con 0x7f9a480a9930 2024-03-29T11:50:03.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.696+0000 7f9a4f796640 10 monclient(hunting): _renew_subs 2024-03-29T11:50:03.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.696+0000 7f9a4f796640 10 monclient(hunting): authenticate will time out at 910.781402s 2024-03-29T11:50:03.704 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.697+0000 7f9a4dd33640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f9a4814ff40 0x7f9a4813a2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51991/0 (socket says 172.21.15.98:51991) 2024-03-29T11:50:03.704 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.698+0000 7f9a4dd33640 1 -- 172.21.15.98:0/3792033036 learned_addr learned my addr 172.21.15.98:0/3792033036 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:50:03.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.699+0000 7f9a3dffb640 10 client.?.objecter ms_handle_connect 0x7f9a4814ff40 2024-03-29T11:50:03.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.699+0000 7f9a3dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:50:03.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.699+0000 7f9a3dffb640 1 -- 172.21.15.98:0/3792033036 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4031571846 0 0) 0x7f9a48072620 con 0x7f9a4814ff40 2024-03-29T11:50:03.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.699+0000 7f9a3dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:50:03.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.699+0000 7f9a3dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:03.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.699+0000 7f9a3dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:03.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.699+0000 7f9a3dffb640 10 monclient(hunting): my global_id is 4287 2024-03-29T11:50:03.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.699+0000 7f9a3dffb640 1 -- 172.21.15.98:0/3792033036 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a2c004eb0 con 0x7f9a4814ff40 2024-03-29T11:50:03.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.699+0000 7f9a3dffb640 1 -- 172.21.15.98:0/3792033036 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2878899432 0 0) 0x7f9a2c004eb0 con 0x7f9a4814ff40 2024-03-29T11:50:03.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.699+0000 7f9a3dffb640 1 -- 172.21.15.98:0/3792033036 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a48072620 con 0x7f9a4814ff40 2024-03-29T11:50:03.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.699+0000 7f9a3dffb640 1 -- 172.21.15.98:0/3792033036 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f9a40004280 con 0x7f9a4814ff40 2024-03-29T11:50:03.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.699+0000 7f9a3dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:03.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.699+0000 7f9a3dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:50:03.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.699+0000 7f9a3dffb640 10 monclient(hunting): dump: 2024-03-29T11:50:03.707 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:50:03.707 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:50:03.707 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:03.707 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:03.707 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:03.707 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:03.707 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:50:03.707 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:50:03.707 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:50:03.708 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:03.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.699+0000 7f9a3dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:50:03.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.699+0000 7f9a3dffb640 1 -- 172.21.15.98:0/3792033036 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1921289397 0 0) 0x7f9a48072620 con 0x7f9a4814ff40 2024-03-29T11:50:03.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.699+0000 7f9a3dffb640 1 -- 172.21.15.98:0/3792033036 >> v1:172.21.15.98:6790/0 conn(0x7f9a480a9930 legacy=0x7f9a4806f830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:50:03.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.700+0000 7f9a3dffb640 1 -- 172.21.15.98:0/3792033036 >> v1:172.21.15.121:6789/0 conn(0x7f9a48149dc0 legacy=0x7f9a48071f10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:50:03.710 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.701+0000 7f9a3dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:50:03.710 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.701+0000 7f9a3dffb640 1 monclient: found mon.a 2024-03-29T11:50:03.710 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.701+0000 7f9a3dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:50:03.710 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.701+0000 7f9a3dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:50:03.710 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.701+0000 7f9a3dffb640 1 -- 172.21.15.98:0/3792033036 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a48071780 con 0x7f9a4814ff40 2024-03-29T11:50:03.710 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.701+0000 7f9a3dffb640 10 monclient: _finish_auth 0 2024-03-29T11:50:03.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.701+0000 7f9a3dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:50:03.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.701+0000 7f9a3dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:50:03.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.701+0000 7f9a3dffb640 1 -- 172.21.15.98:0/3792033036 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a40004850 con 0x7f9a4814ff40 2024-03-29T11:50:03.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.701+0000 7f9a3dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:50:03.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.701+0000 7f9a3dffb640 1 -- 172.21.15.98:0/3792033036 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f9a400013f0 con 0x7f9a4814ff40 2024-03-29T11:50:03.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.701+0000 7f9a3dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:03.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.701+0000 7f9a3dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:50:03.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.701+0000 7f9a3dffb640 10 monclient: dump: 2024-03-29T11:50:03.711 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:50:03.711 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:50:03.711 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:03.711 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:03.711 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:03.711 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:03.711 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:50:03.711 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:50:03.711 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:50:03.712 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:03.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.701+0000 7f9a3f7fe640 4 set_mon_vals no callback set 2024-03-29T11:50:03.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.702+0000 7f9a4f796640 5 monclient: authenticate success, global_id 4287 2024-03-29T11:50:03.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.702+0000 7f9a4f796640 10 monclient: _renew_subs 2024-03-29T11:50:03.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.702+0000 7f9a4f796640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:50:03.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.702+0000 7f9a4f796640 1 -- 172.21.15.98:0/3792033036 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a4806ea90 con 0x7f9a4814ff40 2024-03-29T11:50:03.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.703+0000 7f9a4f796640 10 client.4287.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:50:03.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.703+0000 7f9a4f796640 10 monclient: _renew_subs 2024-03-29T11:50:03.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.703+0000 7f9a4f796640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:50:03.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.703+0000 7f9a4f796640 1 -- 172.21.15.98:0/3792033036 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a4813abb0 con 0x7f9a4814ff40 2024-03-29T11:50:03.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.703+0000 7f9a4f796640 1 librados: init done 2024-03-29T11:50:03.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.705+0000 7f9a3dffb640 1 -- 172.21.15.98:0/3792033036 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f9a40001550 con 0x7f9a4814ff40 2024-03-29T11:50:03.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.705+0000 7f9a4f796640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:03.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.706+0000 7f9a4f796640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:03.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.706+0000 7f9a4f796640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:50:03.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.706+0000 7f9a4f796640 1 -- 172.21.15.98:0/3792033036 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a48070f30 con 0x7f9a4814ff40 2024-03-29T11:50:03.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.706+0000 7f9a3dffb640 1 -- 172.21.15.98:0/3792033036 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f9a40084d40 con 0x7f9a4814ff40 2024-03-29T11:50:03.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.706+0000 7f9a3dffb640 10 client.4287.objecter ms_dispatch 0x7f9a48000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:50:03.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.706+0000 7f9a3dffb640 3 client.4287.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:50:03.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.706+0000 7f9a3dffb640 3 client.4287.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:50:03.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.706+0000 7f9a3dffb640 20 client.4287.objecter dump_active .. 0 homeless 2024-03-29T11:50:03.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.707+0000 7f9a4d532640 10 monclient: get_auth_request con 0x7f9a2c06dc50 auth_method 0 2024-03-29T11:50:03.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.708+0000 7f9a3dffb640 10 client.4287.objecter ms_handle_connect 0x7f9a2c06dc50 2024-03-29T11:50:03.725 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.719+0000 7f9a3dffb640 1 -- 172.21.15.98:0/3792033036 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9a40050620 con 0x7f9a4814ff40 2024-03-29T11:50:03.725 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.719+0000 7f9a3dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:03.725 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:03.719+0000 7f9a3dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:50:04.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:04.037+0000 7f9a4f796640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:04.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:04.037+0000 7f9a4f796640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:04.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:04.037+0000 7f9a4f796640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:50:04.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:04.037+0000 7f9a4f796640 1 -- 172.21.15.98:0/3792033036 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a48071310 con 0x7f9a4814ff40 2024-03-29T11:50:04.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:04.039+0000 7f9a3dffb640 1 -- 172.21.15.98:0/3792033036 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f9a40052550 con 0x7f9a4814ff40 2024-03-29T11:50:04.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:04.039+0000 7f9a3dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:04.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:04.039+0000 7f9a3dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:50:04.048 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:50:04.048 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:50:04.051 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:04.045+0000 7f9a4f796640 10 librados: watch_flush enter 2024-03-29T11:50:04.051 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:04.045+0000 7f9a4f796640 10 librados: watch_flush exit 2024-03-29T11:50:04.051 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:04.045+0000 7f9a4f796640 20 client.4287.objecter shutdown clearing up homeless session... 2024-03-29T11:50:04.051 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:04.045+0000 7f9a4f796640 10 client.4287.objecter successfully canceled tick 2024-03-29T11:50:04.052 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:04.045+0000 7f9a4f796640 1 -- 172.21.15.98:0/3792033036 >> v1:172.21.15.98:6816/4065186974 conn(0x7f9a2c06dc50 legacy=0x7f9a2c070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:04.052 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:04.046+0000 7f9a4f796640 10 monclient: shutdown 2024-03-29T11:50:04.052 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:04.046+0000 7f9a4f796640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:50:04.052 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:04.046+0000 7f9a4f796640 1 -- 172.21.15.98:0/3792033036 >> v1:172.21.15.98:6789/0 conn(0x7f9a4814ff40 legacy=0x7f9a4813a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:04.052 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:04.046+0000 7f9a4f796640 1 -- 172.21.15.98:0/3792033036 shutdown_connections 2024-03-29T11:50:04.052 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:04.046+0000 7f9a4f796640 1 -- 172.21.15.98:0/3792033036 >> 172.21.15.98:0/3792033036 conn(0x7f9a480b35e0 msgr2=0x7f9a480b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:50:04.053 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:04.046+0000 7f9a4f796640 1 -- 172.21.15.98:0/3792033036 shutdown_connections 2024-03-29T11:50:04.053 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:04.046+0000 7f9a4f796640 1 -- 172.21.15.98:0/3792033036 wait complete. 2024-03-29T11:50:04.053 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:04.046+0000 7f9a4f796640 1 librados: shutdown 2024-03-29T11:50:04.072 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:50:10.879 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:50:11.036 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.028+0000 7f5d76b1c640 10 monclient: get_monmap_and_config 2024-03-29T11:50:11.036 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.028+0000 7f5d76b1c640 10 monclient: build_initial_monmap 2024-03-29T11:50:11.036 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.028+0000 7f5d76b1c640 10 monclient: monmap: 2024-03-29T11:50:11.036 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:50:11.036 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:50:11.036 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:50:11.030568+0000 2024-03-29T11:50:11.036 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:50:11.030568+0000 2024-03-29T11:50:11.036 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:50:11.036 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:11.036 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:50:11.036 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:50:11.037 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:50:11.037 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:11.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.030+0000 7f5d76b1c640 1 Processor -- start 2024-03-29T11:50:11.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.030+0000 7f5d76b1c640 1 -- start start 2024-03-29T11:50:11.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.030+0000 7f5d76b1c640 10 monclient: init 2024-03-29T11:50:11.038 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.030+0000 7f5d76b1c640 10 monclient: _reopen_session rank -1 2024-03-29T11:50:11.038 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.030+0000 7f5d76b1c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:50:11.038 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.030+0000 7f5d76b1c640 10 monclient(hunting): picked mon.noname-a con 0x7f5d700b48a0 addr v1:172.21.15.98:6789/0 2024-03-29T11:50:11.038 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.030+0000 7f5d76b1c640 10 monclient(hunting): picked mon.noname-b con 0x7f5d700b3f70 addr v1:172.21.15.98:6790/0 2024-03-29T11:50:11.038 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.030+0000 7f5d76b1c640 10 monclient(hunting): picked mon.noname-c con 0x7f5d7013fb20 addr v1:172.21.15.121:6789/0 2024-03-29T11:50:11.038 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.030+0000 7f5d76b1c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d700b43b0 con 0x7f5d700b48a0 2024-03-29T11:50:11.038 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.031+0000 7f5d76b1c640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d700b0b90 con 0x7f5d7013fb20 2024-03-29T11:50:11.038 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.031+0000 7f5d76b1c640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d700b19a0 con 0x7f5d700b3f70 2024-03-29T11:50:11.038 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.031+0000 7f5d76b1c640 10 monclient(hunting): _renew_subs 2024-03-29T11:50:11.039 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.031+0000 7f5d76b1c640 10 monclient(hunting): authenticate will time out at 918.117078s 2024-03-29T11:50:11.041 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.034+0000 7f5d750b9640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f5d7013fb20 0x7f5d7014da10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:37133/0 (socket says 172.21.15.98:37133) 2024-03-29T11:50:11.041 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.034+0000 7f5d750b9640 1 -- 172.21.15.98:0/1563754593 learned_addr learned my addr 172.21.15.98:0/1563754593 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:50:11.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.035+0000 7f5d6f7fe640 1 -- 172.21.15.98:0/1563754593 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 613002731 0 0) 0x7f5d700b0b90 con 0x7f5d7013fb20 2024-03-29T11:50:11.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.035+0000 7f5d6f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:50:11.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.035+0000 7f5d6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:11.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.035+0000 7f5d6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:11.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.035+0000 7f5d6f7fe640 10 monclient(hunting): my global_id is 4312 2024-03-29T11:50:11.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.035+0000 7f5d6f7fe640 1 -- 172.21.15.98:0/1563754593 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d58003520 con 0x7f5d7013fb20 2024-03-29T11:50:11.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.035+0000 7f5d6f7fe640 1 -- 172.21.15.98:0/1563754593 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 282348837 0 0) 0x7f5d58003520 con 0x7f5d7013fb20 2024-03-29T11:50:11.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.035+0000 7f5d6f7fe640 1 -- 172.21.15.98:0/1563754593 >> v1:172.21.15.98:6790/0 conn(0x7f5d700b3f70 legacy=0x7f5d7013f410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:50:11.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.037+0000 7f5d6f7fe640 1 -- 172.21.15.98:0/1563754593 >> v1:172.21.15.98:6789/0 conn(0x7f5d700b48a0 legacy=0x7f5d700b1290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:50:11.047 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.039+0000 7f5d6f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:50:11.047 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.039+0000 7f5d6f7fe640 1 monclient: found mon.noname-c 2024-03-29T11:50:11.047 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.039+0000 7f5d6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:50:11.047 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.039+0000 7f5d6f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:50:11.047 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.039+0000 7f5d6f7fe640 1 -- 172.21.15.98:0/1563754593 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d700af800 con 0x7f5d7013fb20 2024-03-29T11:50:11.047 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.039+0000 7f5d6f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:50:11.048 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.039+0000 7f5d6f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:50:11.048 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.039+0000 7f5d6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:50:11.048 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.039+0000 7f5d6f7fe640 1 -- 172.21.15.98:0/1563754593 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f5d68001a10 con 0x7f5d7013fb20 2024-03-29T11:50:11.048 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.039+0000 7f5d6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:11.048 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.039+0000 7f5d6f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:50:11.048 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.040+0000 7f5d6f7fe640 10 monclient: dump: 2024-03-29T11:50:11.048 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:50:11.048 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:50:11.048 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:11.048 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:11.048 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:11.048 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:11.048 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:50:11.048 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:50:11.049 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:50:11.049 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:11.049 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.040+0000 7f5d76b1c640 5 monclient: authenticate success, global_id 4312 2024-03-29T11:50:11.049 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.040+0000 7f5d76b1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:50:11.049 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.040+0000 7f5d6f7fe640 1 -- 172.21.15.98:0/1563754593 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d68001ca0 con 0x7f5d7013fb20 2024-03-29T11:50:11.049 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.040+0000 7f5d6f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:50:11.050 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.040+0000 7f5d76b1c640 10 monclient: get_monmap_and_config success 2024-03-29T11:50:11.050 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.040+0000 7f5d76b1c640 4 set_mon_vals no callback set 2024-03-29T11:50:11.050 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.041+0000 7f5d76b1c640 10 monclient: shutdown 2024-03-29T11:50:11.050 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.041+0000 7f5d76b1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:50:11.050 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.041+0000 7f5d76b1c640 1 -- 172.21.15.98:0/1563754593 >> v1:172.21.15.121:6789/0 conn(0x7f5d7013fb20 legacy=0x7f5d7014da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:11.050 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.041+0000 7f5d76b1c640 1 -- 172.21.15.98:0/1563754593 shutdown_connections 2024-03-29T11:50:11.050 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.041+0000 7f5d76b1c640 1 -- 172.21.15.98:0/1563754593 >> 172.21.15.98:0/1563754593 conn(0x7f5d7006c8a0 msgr2=0x7f5d7006cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:50:11.050 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.042+0000 7f5d76b1c640 1 -- 172.21.15.98:0/1563754593 shutdown_connections 2024-03-29T11:50:11.052 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.045+0000 7f5d76b1c640 1 -- 172.21.15.98:0/1563754593 wait complete. 2024-03-29T11:50:11.053 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.046+0000 7f5d76b1c640 10 monclient: build_initial_monmap 2024-03-29T11:50:11.053 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.046+0000 7f5d76b1c640 10 monclient: monmap: 2024-03-29T11:50:11.053 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:50:11.053 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:50:11.053 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:50:11.053 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:50:11.053 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:50:11.053 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:11.053 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:50:11.054 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:50:11.054 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:50:11.054 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:11.054 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.046+0000 7f5d76b1c640 1 librados: starting msgr at 2024-03-29T11:50:11.055 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.046+0000 7f5d76b1c640 1 librados: starting objecter 2024-03-29T11:50:11.055 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.046+0000 7f5d76b1c640 1 Processor -- start 2024-03-29T11:50:11.055 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.047+0000 7f5d76b1c640 1 -- start start 2024-03-29T11:50:11.055 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.047+0000 7f5d76b1c640 1 librados: setting wanted keys 2024-03-29T11:50:11.055 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.047+0000 7f5d76b1c640 1 librados: calling monclient init 2024-03-29T11:50:11.055 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.047+0000 7f5d76b1c640 10 monclient: init 2024-03-29T11:50:11.057 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.050+0000 7f5d76b1c640 10 monclient: _reopen_session rank -1 2024-03-29T11:50:11.057 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.050+0000 7f5d76b1c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:50:11.057 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.050+0000 7f5d76b1c640 10 monclient(hunting): picked mon.noname-b con 0x7f5d700b3f70 addr v1:172.21.15.121:6789/0 2024-03-29T11:50:11.057 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.050+0000 7f5d76b1c640 10 monclient(hunting): picked mon.noname-c con 0x7f5d700b48a0 addr v1:172.21.15.98:6790/0 2024-03-29T11:50:11.057 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.050+0000 7f5d76b1c640 10 monclient(hunting): picked mon.noname-a con 0x7f5d7013fb20 addr v1:172.21.15.98:6789/0 2024-03-29T11:50:11.058 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.050+0000 7f5d76b1c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d70071a70 con 0x7f5d7013fb20 2024-03-29T11:50:11.058 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.050+0000 7f5d76b1c640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d7006eea0 con 0x7f5d700b3f70 2024-03-29T11:50:11.058 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.050+0000 7f5d76b1c640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d70070c60 con 0x7f5d700b48a0 2024-03-29T11:50:11.058 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.050+0000 7f5d76b1c640 10 monclient(hunting): _renew_subs 2024-03-29T11:50:11.058 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.050+0000 7f5d76b1c640 10 monclient(hunting): authenticate will time out at 918.136486s 2024-03-29T11:50:11.067 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.057+0000 7f5d750b9640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f5d7013fb20 0x7f5d701e60b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:41253/0 (socket says 172.21.15.98:41253) 2024-03-29T11:50:11.067 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.057+0000 7f5d750b9640 1 -- 172.21.15.98:0/3809271760 learned_addr learned my addr 172.21.15.98:0/3809271760 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:50:11.067 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.057+0000 7f5d6d7fa640 10 client.?.objecter ms_handle_connect 0x7f5d7013fb20 2024-03-29T11:50:11.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.057+0000 7f5d6d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:50:11.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.057+0000 7f5d6d7fa640 10 client.?.objecter ms_handle_connect 0x7f5d700b48a0 2024-03-29T11:50:11.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.057+0000 7f5d6d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:50:11.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.057+0000 7f5d6d7fa640 10 client.?.objecter ms_handle_connect 0x7f5d700b3f70 2024-03-29T11:50:11.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.057+0000 7f5d6d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:50:11.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.057+0000 7f5d6d7fa640 1 -- 172.21.15.98:0/3809271760 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2764562246 0 0) 0x7f5d70071a70 con 0x7f5d7013fb20 2024-03-29T11:50:11.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.057+0000 7f5d6d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:50:11.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.057+0000 7f5d6d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:11.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.057+0000 7f5d6d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:11.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.057+0000 7f5d6d7fa640 10 monclient(hunting): my global_id is 4290 2024-03-29T11:50:11.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.058+0000 7f5d6d7fa640 1 -- 172.21.15.98:0/3809271760 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d5c004eb0 con 0x7f5d7013fb20 2024-03-29T11:50:11.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.058+0000 7f5d6d7fa640 1 -- 172.21.15.98:0/3809271760 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 250161266 0 0) 0x7f5d70070c60 con 0x7f5d700b48a0 2024-03-29T11:50:11.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.058+0000 7f5d6d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:50:11.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.058+0000 7f5d6d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:11.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.058+0000 7f5d6d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:11.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.058+0000 7f5d6d7fa640 10 monclient(hunting): my global_id is 4298 2024-03-29T11:50:11.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.058+0000 7f5d6d7fa640 1 -- 172.21.15.98:0/3809271760 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d70071a70 con 0x7f5d700b48a0 2024-03-29T11:50:11.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.058+0000 7f5d6d7fa640 1 -- 172.21.15.98:0/3809271760 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2863860915 0 0) 0x7f5d7006eea0 con 0x7f5d700b3f70 2024-03-29T11:50:11.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.058+0000 7f5d6d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:50:11.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.058+0000 7f5d6d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:11.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.059+0000 7f5d6d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:11.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.059+0000 7f5d6d7fa640 10 monclient(hunting): my global_id is 4315 2024-03-29T11:50:11.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.059+0000 7f5d6d7fa640 1 -- 172.21.15.98:0/3809271760 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d70070c60 con 0x7f5d700b3f70 2024-03-29T11:50:11.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.059+0000 7f5d6d7fa640 1 -- 172.21.15.98:0/3809271760 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2208852199 0 0) 0x7f5d5c004eb0 con 0x7f5d7013fb20 2024-03-29T11:50:11.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.059+0000 7f5d6d7fa640 1 -- 172.21.15.98:0/3809271760 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d7006eea0 con 0x7f5d7013fb20 2024-03-29T11:50:11.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.059+0000 7f5d6d7fa640 1 -- 172.21.15.98:0/3809271760 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f5d68004e00 con 0x7f5d7013fb20 2024-03-29T11:50:11.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.059+0000 7f5d6d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:11.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.060+0000 7f5d6d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:50:11.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.060+0000 7f5d6d7fa640 10 monclient(hunting): dump: 2024-03-29T11:50:11.069 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:50:11.069 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:50:11.070 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:11.070 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:11.070 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:11.070 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:11.070 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:50:11.070 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:50:11.070 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:50:11.070 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:11.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.060+0000 7f5d6d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:50:11.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.060+0000 7f5d6d7fa640 1 -- 172.21.15.98:0/3809271760 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1883734466 0 0) 0x7f5d70070c60 con 0x7f5d700b3f70 2024-03-29T11:50:11.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.060+0000 7f5d6d7fa640 1 -- 172.21.15.98:0/3809271760 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d5c004eb0 con 0x7f5d700b3f70 2024-03-29T11:50:11.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.060+0000 7f5d6d7fa640 1 -- 172.21.15.98:0/3809271760 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3554870724 0 0) 0x7f5d7006eea0 con 0x7f5d7013fb20 2024-03-29T11:50:11.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.060+0000 7f5d6d7fa640 1 -- 172.21.15.98:0/3809271760 >> v1:172.21.15.98:6790/0 conn(0x7f5d700b48a0 legacy=0x7f5d701458c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:11.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.060+0000 7f5d6d7fa640 1 -- 172.21.15.98:0/3809271760 >> v1:172.21.15.121:6789/0 conn(0x7f5d700b3f70 legacy=0x7f5d70071360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:11.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.061+0000 7f5d6d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:50:11.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.061+0000 7f5d6d7fa640 1 monclient: found mon.a 2024-03-29T11:50:11.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.061+0000 7f5d6d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:50:11.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.061+0000 7f5d6d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:50:11.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.061+0000 7f5d6d7fa640 1 -- 172.21.15.98:0/3809271760 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d70070f40 con 0x7f5d7013fb20 2024-03-29T11:50:11.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.061+0000 7f5d6d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:50:11.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.061+0000 7f5d6d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:50:11.072 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.061+0000 7f5d6d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:50:11.072 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.061+0000 7f5d76b1c640 5 monclient: authenticate success, global_id 4290 2024-03-29T11:50:11.072 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.061+0000 7f5d76b1c640 10 monclient: _renew_subs 2024-03-29T11:50:11.072 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.061+0000 7f5d76b1c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:50:11.072 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.061+0000 7f5d76b1c640 1 -- 172.21.15.98:0/3809271760 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d7006e980 con 0x7f5d7013fb20 2024-03-29T11:50:11.072 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.061+0000 7f5d76b1c640 10 client.4290.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:50:11.072 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.061+0000 7f5d76b1c640 10 monclient: _renew_subs 2024-03-29T11:50:11.072 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.061+0000 7f5d76b1c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:50:11.072 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.061+0000 7f5d76b1c640 1 -- 172.21.15.98:0/3809271760 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d7006de90 con 0x7f5d7013fb20 2024-03-29T11:50:11.072 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.061+0000 7f5d76b1c640 1 librados: init done 2024-03-29T11:50:11.072 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.061+0000 7f5d6d7fa640 1 -- 172.21.15.98:0/3809271760 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d68001360 con 0x7f5d7013fb20 2024-03-29T11:50:11.072 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.061+0000 7f5d6d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:50:11.072 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.061+0000 7f5d6d7fa640 1 -- 172.21.15.98:0/3809271760 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f5d68002070 con 0x7f5d7013fb20 2024-03-29T11:50:11.072 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.061+0000 7f5d6d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:11.072 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.061+0000 7f5d6d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:50:11.072 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.061+0000 7f5d6d7fa640 10 monclient: dump: 2024-03-29T11:50:11.073 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:50:11.073 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:50:11.073 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:11.073 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:11.073 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:11.073 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:11.073 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:50:11.073 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:50:11.073 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:50:11.073 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:11.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.061+0000 7f5d6effd640 4 set_mon_vals no callback set 2024-03-29T11:50:11.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.063+0000 7f5d6d7fa640 1 -- 172.21.15.98:0/3809271760 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f5d68005210 con 0x7f5d7013fb20 2024-03-29T11:50:11.074 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.064+0000 7f5d76b1c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:11.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.064+0000 7f5d76b1c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:11.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.064+0000 7f5d76b1c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:50:11.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.064+0000 7f5d76b1c640 1 -- 172.21.15.98:0/3809271760 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d50005080 con 0x7f5d7013fb20 2024-03-29T11:50:11.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.065+0000 7f5d6d7fa640 1 -- 172.21.15.98:0/3809271760 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f5d68084f70 con 0x7f5d7013fb20 2024-03-29T11:50:11.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.065+0000 7f5d6d7fa640 10 client.4290.objecter ms_dispatch 0x7f5d70000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:50:11.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.065+0000 7f5d6d7fa640 3 client.4290.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:50:11.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.065+0000 7f5d6d7fa640 3 client.4290.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:50:11.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.065+0000 7f5d6d7fa640 20 client.4290.objecter dump_active .. 0 homeless 2024-03-29T11:50:11.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.066+0000 7f5d748b8640 10 monclient: get_auth_request con 0x7f5d5c06dff0 auth_method 0 2024-03-29T11:50:11.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.066+0000 7f5d6d7fa640 10 client.4290.objecter ms_handle_connect 0x7f5d5c06dff0 2024-03-29T11:50:11.085 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.077+0000 7f5d6d7fa640 1 -- 172.21.15.98:0/3809271760 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5d68050850 con 0x7f5d7013fb20 2024-03-29T11:50:11.085 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.077+0000 7f5d6d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:11.085 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.077+0000 7f5d6d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:50:11.399 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.392+0000 7f5d76b1c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:11.399 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.392+0000 7f5d76b1c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:11.399 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.392+0000 7f5d76b1c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:50:11.400 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.392+0000 7f5d76b1c640 1 -- 172.21.15.98:0/3809271760 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d50005330 con 0x7f5d7013fb20 2024-03-29T11:50:11.400 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.393+0000 7f5d6d7fa640 1 -- 172.21.15.98:0/3809271760 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f5d68052780 con 0x7f5d7013fb20 2024-03-29T11:50:11.400 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.393+0000 7f5d6d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:11.400 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.393+0000 7f5d6d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:50:11.402 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:50:11.403 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:50:11.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.398+0000 7f5d76b1c640 10 librados: watch_flush enter 2024-03-29T11:50:11.407 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.400+0000 7f5d76b1c640 10 librados: watch_flush exit 2024-03-29T11:50:11.407 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.400+0000 7f5d76b1c640 20 client.4290.objecter shutdown clearing up homeless session... 2024-03-29T11:50:11.407 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.400+0000 7f5d76b1c640 10 client.4290.objecter successfully canceled tick 2024-03-29T11:50:11.408 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.400+0000 7f5d76b1c640 1 -- 172.21.15.98:0/3809271760 >> v1:172.21.15.98:6816/4065186974 conn(0x7f5d5c06dff0 legacy=0x7f5d5c0704b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:11.408 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.400+0000 7f5d76b1c640 10 monclient: shutdown 2024-03-29T11:50:11.408 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.400+0000 7f5d76b1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:50:11.408 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.400+0000 7f5d76b1c640 1 -- 172.21.15.98:0/3809271760 >> v1:172.21.15.98:6789/0 conn(0x7f5d7013fb20 legacy=0x7f5d701e60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:11.408 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.401+0000 7f5d76b1c640 1 -- 172.21.15.98:0/3809271760 shutdown_connections 2024-03-29T11:50:11.408 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.401+0000 7f5d76b1c640 1 -- 172.21.15.98:0/3809271760 >> 172.21.15.98:0/3809271760 conn(0x7f5d7006c8a0 msgr2=0x7f5d700b11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:50:11.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.403+0000 7f5d76b1c640 1 -- 172.21.15.98:0/3809271760 shutdown_connections 2024-03-29T11:50:11.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.403+0000 7f5d76b1c640 1 -- 172.21.15.98:0/3809271760 wait complete. 2024-03-29T11:50:11.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:11.403+0000 7f5d76b1c640 1 librados: shutdown 2024-03-29T11:50:11.428 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:50:18.235 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:50:18.384 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.377+0000 7fa874899640 10 monclient: get_monmap_and_config 2024-03-29T11:50:18.384 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.377+0000 7fa874899640 10 monclient: build_initial_monmap 2024-03-29T11:50:18.384 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.377+0000 7fa874899640 10 monclient: monmap: 2024-03-29T11:50:18.385 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:50:18.385 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:50:18.385 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:50:18.379217+0000 2024-03-29T11:50:18.385 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:50:18.379217+0000 2024-03-29T11:50:18.385 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:50:18.385 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:18.385 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:50:18.385 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:50:18.385 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:50:18.385 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:18.386 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.379+0000 7fa874899640 1 Processor -- start 2024-03-29T11:50:18.386 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.379+0000 7fa874899640 1 -- start start 2024-03-29T11:50:18.386 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.379+0000 7fa874899640 10 monclient: init 2024-03-29T11:50:18.391 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.380+0000 7fa874899640 10 monclient: _reopen_session rank -1 2024-03-29T11:50:18.391 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.380+0000 7fa874899640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:50:18.391 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.380+0000 7fa874899640 10 monclient(hunting): picked mon.noname-b con 0x7fa868006a70 addr v1:172.21.15.98:6790/0 2024-03-29T11:50:18.392 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.380+0000 7fa874899640 10 monclient(hunting): picked mon.noname-c con 0x7fa86802b100 addr v1:172.21.15.121:6789/0 2024-03-29T11:50:18.392 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.380+0000 7fa874899640 10 monclient(hunting): picked mon.noname-a con 0x7fa868029c30 addr v1:172.21.15.98:6789/0 2024-03-29T11:50:18.392 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.380+0000 7fa874899640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa86802a270 con 0x7fa868029c30 2024-03-29T11:50:18.392 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.380+0000 7fa874899640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa86802c440 con 0x7fa86802b100 2024-03-29T11:50:18.392 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.380+0000 7fa874899640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8680c9440 con 0x7fa868006a70 2024-03-29T11:50:18.392 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.380+0000 7fa874899640 10 monclient(hunting): _renew_subs 2024-03-29T11:50:18.392 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.380+0000 7fa874899640 10 monclient(hunting): authenticate will time out at 925.466193s 2024-03-29T11:50:18.392 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.383+0000 7fa86e59c640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fa86802b100 0x7fa8680cf100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:46891/0 (socket says 172.21.15.98:46891) 2024-03-29T11:50:18.392 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.383+0000 7fa86e59c640 1 -- 172.21.15.98:0/2549393677 learned_addr learned my addr 172.21.15.98:0/2549393677 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:50:18.392 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.384+0000 7fa86dd9b640 1 -- 172.21.15.98:0/2549393677 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2163003091 0 0) 0x7fa86802c440 con 0x7fa86802b100 2024-03-29T11:50:18.392 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.384+0000 7fa86dd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:50:18.392 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.384+0000 7fa86dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:18.392 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.384+0000 7fa86dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:18.393 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.384+0000 7fa86dd9b640 10 monclient(hunting): my global_id is 4318 2024-03-29T11:50:18.393 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.384+0000 7fa86dd9b640 1 -- 172.21.15.98:0/2549393677 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa858003520 con 0x7fa86802b100 2024-03-29T11:50:18.393 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.384+0000 7fa86dd9b640 1 -- 172.21.15.98:0/2549393677 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2139987463 0 0) 0x7fa858003520 con 0x7fa86802b100 2024-03-29T11:50:18.393 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.384+0000 7fa86dd9b640 1 -- 172.21.15.98:0/2549393677 >> v1:172.21.15.98:6790/0 conn(0x7fa868006a70 legacy=0x7fa8680c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:50:18.393 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.384+0000 7fa86dd9b640 1 -- 172.21.15.98:0/2549393677 >> v1:172.21.15.98:6789/0 conn(0x7fa868029c30 legacy=0x7fa8680d74a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:50:18.395 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.387+0000 7fa86dd9b640 10 monclient: _finish_hunting 0 2024-03-29T11:50:18.395 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.387+0000 7fa86dd9b640 1 monclient: found mon.noname-c 2024-03-29T11:50:18.395 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.387+0000 7fa86dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:50:18.395 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.387+0000 7fa86dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:50:18.396 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.387+0000 7fa86dd9b640 1 -- 172.21.15.98:0/2549393677 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8680c95d0 con 0x7fa86802b100 2024-03-29T11:50:18.396 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.387+0000 7fa86dd9b640 10 monclient: _finish_auth 0 2024-03-29T11:50:18.396 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.387+0000 7fa86dd9b640 10 monclient: _check_auth_tickets 2024-03-29T11:50:18.396 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.387+0000 7fa86dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:50:18.396 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.387+0000 7fa874899640 5 monclient: authenticate success, global_id 4318 2024-03-29T11:50:18.396 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.387+0000 7fa874899640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:50:18.396 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.387+0000 7fa86dd9b640 1 -- 172.21.15.98:0/2549393677 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fa860001690 con 0x7fa86802b100 2024-03-29T11:50:18.396 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.387+0000 7fa86dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:18.396 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.387+0000 7fa86dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:50:18.396 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.387+0000 7fa86dd9b640 10 monclient: dump: 2024-03-29T11:50:18.396 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:50:18.396 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:50:18.396 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:18.396 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:18.397 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:18.397 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:18.397 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:50:18.397 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:50:18.397 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:50:18.397 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:18.397 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.387+0000 7fa874899640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:50:18.398 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.388+0000 7fa86dd9b640 1 -- 172.21.15.98:0/2549393677 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8600019a0 con 0x7fa86802b100 2024-03-29T11:50:18.398 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.388+0000 7fa86dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:50:18.398 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.388+0000 7fa874899640 10 monclient: get_monmap_and_config success 2024-03-29T11:50:18.398 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.388+0000 7fa874899640 4 set_mon_vals no callback set 2024-03-29T11:50:18.398 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.389+0000 7fa874899640 10 monclient: shutdown 2024-03-29T11:50:18.398 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.390+0000 7fa874899640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:50:18.398 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.390+0000 7fa874899640 1 -- 172.21.15.98:0/2549393677 >> v1:172.21.15.121:6789/0 conn(0x7fa86802b100 legacy=0x7fa8680cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:18.398 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.390+0000 7fa874899640 1 -- 172.21.15.98:0/2549393677 shutdown_connections 2024-03-29T11:50:18.399 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.390+0000 7fa874899640 1 -- 172.21.15.98:0/2549393677 >> 172.21.15.98:0/2549393677 conn(0x7fa8680058f0 msgr2=0x7fa868005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:50:18.399 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.391+0000 7fa874899640 1 -- 172.21.15.98:0/2549393677 shutdown_connections 2024-03-29T11:50:18.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.391+0000 7fa874899640 1 -- 172.21.15.98:0/2549393677 wait complete. 2024-03-29T11:50:18.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.392+0000 7fa874899640 10 monclient: build_initial_monmap 2024-03-29T11:50:18.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.392+0000 7fa874899640 10 monclient: monmap: 2024-03-29T11:50:18.405 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:50:18.405 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:50:18.405 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:50:18.405 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:50:18.405 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:50:18.405 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:18.405 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:50:18.405 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:50:18.405 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:50:18.405 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:18.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.393+0000 7fa874899640 1 librados: starting msgr at 2024-03-29T11:50:18.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.393+0000 7fa874899640 1 librados: starting objecter 2024-03-29T11:50:18.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.393+0000 7fa874899640 1 Processor -- start 2024-03-29T11:50:18.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.393+0000 7fa874899640 1 -- start start 2024-03-29T11:50:18.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.393+0000 7fa874899640 1 librados: setting wanted keys 2024-03-29T11:50:18.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.393+0000 7fa874899640 1 librados: calling monclient init 2024-03-29T11:50:18.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.393+0000 7fa874899640 10 monclient: init 2024-03-29T11:50:18.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.394+0000 7fa874899640 10 monclient: _reopen_session rank -1 2024-03-29T11:50:18.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.394+0000 7fa874899640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:50:18.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.394+0000 7fa874899640 10 monclient(hunting): picked mon.noname-c con 0x7fa868006a70 addr v1:172.21.15.98:6790/0 2024-03-29T11:50:18.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.394+0000 7fa874899640 10 monclient(hunting): picked mon.noname-a con 0x7fa868029c30 addr v1:172.21.15.98:6789/0 2024-03-29T11:50:18.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.394+0000 7fa874899640 10 monclient(hunting): picked mon.noname-b con 0x7fa8680ca490 addr v1:172.21.15.121:6789/0 2024-03-29T11:50:18.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.394+0000 7fa874899640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8680caaf0 con 0x7fa868029c30 2024-03-29T11:50:18.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.394+0000 7fa874899640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa868186050 con 0x7fa8680ca490 2024-03-29T11:50:18.407 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.394+0000 7fa874899640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8680c9980 con 0x7fa868006a70 2024-03-29T11:50:18.407 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.394+0000 7fa874899640 10 monclient(hunting): _renew_subs 2024-03-29T11:50:18.407 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.394+0000 7fa874899640 10 monclient(hunting): authenticate will time out at 925.480392s 2024-03-29T11:50:18.407 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.399+0000 7fa86e59c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa868029c30 0x7fa868031c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38515/0 (socket says 172.21.15.98:38515) 2024-03-29T11:50:18.407 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.399+0000 7fa86e59c640 1 -- 172.21.15.98:0/382109374 learned_addr learned my addr 172.21.15.98:0/382109374 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:50:18.408 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.399+0000 7fa8577fe640 10 client.?.objecter ms_handle_connect 0x7fa868029c30 2024-03-29T11:50:18.408 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.399+0000 7fa8577fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:50:18.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.401+0000 7fa8577fe640 10 client.?.objecter ms_handle_connect 0x7fa868006a70 2024-03-29T11:50:18.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.401+0000 7fa8577fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:50:18.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.401+0000 7fa8577fe640 1 -- 172.21.15.98:0/382109374 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1573844844 0 0) 0x7fa8680c9980 con 0x7fa868006a70 2024-03-29T11:50:18.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.401+0000 7fa8577fe640 10 monclient(hunting): _negotiate 2024-03-29T11:50:18.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.401+0000 7fa8577fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:18.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.401+0000 7fa8577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:18.417 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.401+0000 7fa8577fe640 10 monclient(hunting): my global_id is 4301 2024-03-29T11:50:18.417 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.401+0000 7fa8577fe640 1 -- 172.21.15.98:0/382109374 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa85c004eb0 con 0x7fa868006a70 2024-03-29T11:50:18.417 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.401+0000 7fa8577fe640 1 -- 172.21.15.98:0/382109374 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1635003545 0 0) 0x7fa85c004eb0 con 0x7fa868006a70 2024-03-29T11:50:18.417 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.401+0000 7fa8577fe640 1 -- 172.21.15.98:0/382109374 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8680c9980 con 0x7fa868006a70 2024-03-29T11:50:18.417 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.401+0000 7fa8577fe640 1 -- 172.21.15.98:0/382109374 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fa87004fdd0 con 0x7fa868006a70 2024-03-29T11:50:18.417 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.401+0000 7fa8577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:18.417 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.401+0000 7fa8577fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:50:18.417 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.401+0000 7fa8577fe640 10 monclient(hunting): dump: 2024-03-29T11:50:18.417 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:50:18.417 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:50:18.417 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:18.417 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:18.417 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:18.417 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:18.418 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:50:18.418 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:50:18.418 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:50:18.418 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:18.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.401+0000 7fa8577fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:50:18.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.402+0000 7fa8577fe640 1 -- 172.21.15.98:0/382109374 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3183808272 0 0) 0x7fa8680c9980 con 0x7fa868006a70 2024-03-29T11:50:18.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.402+0000 7fa8577fe640 1 -- 172.21.15.98:0/382109374 >> v1:172.21.15.121:6789/0 conn(0x7fa8680ca490 legacy=0x7fa86818e230 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:50:18.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.402+0000 7fa8577fe640 1 -- 172.21.15.98:0/382109374 >> v1:172.21.15.98:6789/0 conn(0x7fa868029c30 legacy=0x7fa868031c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:18.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.402+0000 7fa8577fe640 10 monclient: _finish_hunting 0 2024-03-29T11:50:18.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.402+0000 7fa8577fe640 1 monclient: found mon.c 2024-03-29T11:50:18.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.402+0000 7fa8577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:50:18.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.402+0000 7fa8577fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:50:18.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.402+0000 7fa8577fe640 1 -- 172.21.15.98:0/382109374 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8680c9b10 con 0x7fa868006a70 2024-03-29T11:50:18.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.402+0000 7fa8577fe640 10 monclient: _finish_auth 0 2024-03-29T11:50:18.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.402+0000 7fa8577fe640 10 monclient: _check_auth_tickets 2024-03-29T11:50:18.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.402+0000 7fa8577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:50:18.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.402+0000 7fa874899640 5 monclient: authenticate success, global_id 4301 2024-03-29T11:50:18.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.402+0000 7fa874899640 10 monclient: _renew_subs 2024-03-29T11:50:18.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.402+0000 7fa874899640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:50:18.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.402+0000 7fa874899640 1 -- 172.21.15.98:0/382109374 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8680334c0 con 0x7fa868006a70 2024-03-29T11:50:18.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.402+0000 7fa874899640 10 client.4301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:50:18.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.402+0000 7fa874899640 10 monclient: _renew_subs 2024-03-29T11:50:18.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.402+0000 7fa874899640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:50:18.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.402+0000 7fa874899640 1 -- 172.21.15.98:0/382109374 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa868186dd0 con 0x7fa868006a70 2024-03-29T11:50:18.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.402+0000 7fa874899640 1 librados: init done 2024-03-29T11:50:18.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.402+0000 7fa8577fe640 1 -- 172.21.15.98:0/382109374 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa87004f400 con 0x7fa868006a70 2024-03-29T11:50:18.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.402+0000 7fa8577fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:50:18.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.402+0000 7fa8577fe640 1 -- 172.21.15.98:0/382109374 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fa870075f10 con 0x7fa868006a70 2024-03-29T11:50:18.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.402+0000 7fa8577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:18.420 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.402+0000 7fa8577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:50:18.420 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.402+0000 7fa8577fe640 10 monclient: dump: 2024-03-29T11:50:18.420 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:50:18.420 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:50:18.420 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:18.420 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:18.420 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:18.420 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:18.420 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:50:18.420 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:50:18.420 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:50:18.420 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:18.420 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.403+0000 7fa86cd99640 4 set_mon_vals no callback set 2024-03-29T11:50:18.420 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.403+0000 7fa874899640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:18.420 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.403+0000 7fa874899640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:18.420 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.403+0000 7fa874899640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:50:18.421 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.403+0000 7fa874899640 1 -- 172.21.15.98:0/382109374 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa850005080 con 0x7fa868006a70 2024-03-29T11:50:18.421 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.405+0000 7fa8577fe640 1 -- 172.21.15.98:0/382109374 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7fa870067bf0 con 0x7fa868006a70 2024-03-29T11:50:18.421 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.406+0000 7fa86e59c640 10 monclient: get_auth_request con 0x7fa85c06dbc0 auth_method 0 2024-03-29T11:50:18.421 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.406+0000 7fa8577fe640 10 client.4301.objecter ms_handle_connect 0x7fa85c06dbc0 2024-03-29T11:50:18.421 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.406+0000 7fa8577fe640 1 -- 172.21.15.98:0/382109374 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7fa8700ed260 con 0x7fa868006a70 2024-03-29T11:50:18.421 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.406+0000 7fa8577fe640 10 client.4301.objecter ms_dispatch 0x7fa870000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:50:18.421 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.406+0000 7fa8577fe640 3 client.4301.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:50:18.421 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.406+0000 7fa8577fe640 3 client.4301.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:50:18.421 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.406+0000 7fa8577fe640 20 client.4301.objecter dump_active .. 0 homeless 2024-03-29T11:50:18.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.415+0000 7fa8577fe640 1 -- 172.21.15.98:0/382109374 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa8700b8b60 con 0x7fa868006a70 2024-03-29T11:50:18.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.415+0000 7fa8577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:18.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.415+0000 7fa8577fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:50:18.755 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.749+0000 7fa874899640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:18.755 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.749+0000 7fa874899640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:18.755 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.749+0000 7fa874899640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:50:18.755 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.749+0000 7fa874899640 1 -- 172.21.15.98:0/382109374 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa850005330 con 0x7fa868006a70 2024-03-29T11:50:18.756 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.750+0000 7fa8577fe640 1 -- 172.21.15.98:0/382109374 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7fa8700baa90 con 0x7fa868006a70 2024-03-29T11:50:18.756 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.750+0000 7fa8577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:18.756 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.750+0000 7fa8577fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:50:18.758 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:50:18.758 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:50:18.762 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.756+0000 7fa874899640 10 librados: watch_flush enter 2024-03-29T11:50:18.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.756+0000 7fa874899640 10 librados: watch_flush exit 2024-03-29T11:50:18.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.756+0000 7fa874899640 20 client.4301.objecter shutdown clearing up homeless session... 2024-03-29T11:50:18.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.756+0000 7fa874899640 10 client.4301.objecter successfully canceled tick 2024-03-29T11:50:18.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.756+0000 7fa874899640 1 -- 172.21.15.98:0/382109374 >> v1:172.21.15.98:6816/4065186974 conn(0x7fa85c06dbc0 legacy=0x7fa85c070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:18.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.756+0000 7fa874899640 10 monclient: shutdown 2024-03-29T11:50:18.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.756+0000 7fa874899640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:50:18.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.756+0000 7fa874899640 1 -- 172.21.15.98:0/382109374 >> v1:172.21.15.98:6790/0 conn(0x7fa868006a70 legacy=0x7fa8681864e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:18.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.756+0000 7fa874899640 1 -- 172.21.15.98:0/382109374 shutdown_connections 2024-03-29T11:50:18.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.756+0000 7fa874899640 1 -- 172.21.15.98:0/382109374 >> 172.21.15.98:0/382109374 conn(0x7fa8680058f0 msgr2=0x7fa8680c3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:50:18.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.756+0000 7fa874899640 1 -- 172.21.15.98:0/382109374 shutdown_connections 2024-03-29T11:50:18.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.756+0000 7fa874899640 1 -- 172.21.15.98:0/382109374 wait complete. 2024-03-29T11:50:18.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:18.756+0000 7fa874899640 1 librados: shutdown 2024-03-29T11:50:18.781 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:50:25.588 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:50:25.738 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.732+0000 7f08b49e1640 10 monclient: get_monmap_and_config 2024-03-29T11:50:25.739 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.732+0000 7f08b49e1640 10 monclient: build_initial_monmap 2024-03-29T11:50:25.739 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.732+0000 7f08b49e1640 10 monclient: monmap: 2024-03-29T11:50:25.739 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:50:25.739 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:50:25.739 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:50:25.733224+0000 2024-03-29T11:50:25.739 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:50:25.733224+0000 2024-03-29T11:50:25.739 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:50:25.739 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:25.739 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:50:25.739 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:50:25.739 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:50:25.739 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:25.741 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.733+0000 7f08b49e1640 1 Processor -- start 2024-03-29T11:50:25.741 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.733+0000 7f08b49e1640 1 -- start start 2024-03-29T11:50:25.741 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.733+0000 7f08b49e1640 10 monclient: init 2024-03-29T11:50:25.741 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.734+0000 7f08b49e1640 10 monclient: _reopen_session rank -1 2024-03-29T11:50:25.741 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.734+0000 7f08b49e1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:50:25.742 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.734+0000 7f08b49e1640 10 monclient(hunting): picked mon.noname-c con 0x7f08a8038c60 addr v1:172.21.15.121:6789/0 2024-03-29T11:50:25.742 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.734+0000 7f08b49e1640 10 monclient(hunting): picked mon.noname-b con 0x7f08a80324b0 addr v1:172.21.15.98:6790/0 2024-03-29T11:50:25.742 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.734+0000 7f08b49e1640 10 monclient(hunting): picked mon.noname-a con 0x7f08a80d11e0 addr v1:172.21.15.98:6789/0 2024-03-29T11:50:25.742 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.734+0000 7f08b49e1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08a80caf80 con 0x7f08a80d11e0 2024-03-29T11:50:25.742 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.734+0000 7f08b49e1640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08a8039850 con 0x7f08a8038c60 2024-03-29T11:50:25.742 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.734+0000 7f08b49e1640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08a80c78d0 con 0x7f08a80324b0 2024-03-29T11:50:25.742 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.734+0000 7f08b49e1640 10 monclient(hunting): _renew_subs 2024-03-29T11:50:25.742 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.734+0000 7f08b49e1640 10 monclient(hunting): authenticate will time out at 932.819585s 2024-03-29T11:50:25.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.737+0000 7f08af7fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f08a8038c60 0x7f08a80c4ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:36667/0 (socket says 172.21.15.98:36667) 2024-03-29T11:50:25.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.737+0000 7f08af7fe640 1 -- 172.21.15.98:0/2992993955 learned_addr learned my addr 172.21.15.98:0/2992993955 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:50:25.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.740+0000 7f08ae7fc640 1 -- 172.21.15.98:0/2992993955 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3880172807 0 0) 0x7f08a80caf80 con 0x7f08a80d11e0 2024-03-29T11:50:25.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.740+0000 7f08ae7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:50:25.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.740+0000 7f08ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:25.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.740+0000 7f08ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:25.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.740+0000 7f08ae7fc640 10 monclient(hunting): my global_id is 4296 2024-03-29T11:50:25.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.740+0000 7f08ae7fc640 1 -- 172.21.15.98:0/2992993955 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08a0003520 con 0x7f08a80d11e0 2024-03-29T11:50:25.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.740+0000 7f08ae7fc640 1 -- 172.21.15.98:0/2992993955 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3381541399 0 0) 0x7f08a80c78d0 con 0x7f08a80324b0 2024-03-29T11:50:25.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.740+0000 7f08ae7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:50:25.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.740+0000 7f08ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:25.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.740+0000 7f08ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:25.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.740+0000 7f08ae7fc640 10 monclient(hunting): my global_id is 4304 2024-03-29T11:50:25.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.740+0000 7f08ae7fc640 1 -- 172.21.15.98:0/2992993955 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08a80caf80 con 0x7f08a80324b0 2024-03-29T11:50:25.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.740+0000 7f08ae7fc640 1 -- 172.21.15.98:0/2992993955 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1419701750 0 0) 0x7f08a8039850 con 0x7f08a8038c60 2024-03-29T11:50:25.750 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.740+0000 7f08ae7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:50:25.750 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.740+0000 7f08ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:25.750 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.740+0000 7f08ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:25.750 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.740+0000 7f08ae7fc640 10 monclient(hunting): my global_id is 4321 2024-03-29T11:50:25.750 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.740+0000 7f08ae7fc640 1 -- 172.21.15.98:0/2992993955 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08a80c78d0 con 0x7f08a8038c60 2024-03-29T11:50:25.750 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.740+0000 7f08ae7fc640 1 -- 172.21.15.98:0/2992993955 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3822424481 0 0) 0x7f08a80caf80 con 0x7f08a80324b0 2024-03-29T11:50:25.750 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.740+0000 7f08ae7fc640 1 -- 172.21.15.98:0/2992993955 >> v1:172.21.15.121:6789/0 conn(0x7f08a8038c60 legacy=0x7f08a80c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:25.750 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.740+0000 7f08ae7fc640 1 -- 172.21.15.98:0/2992993955 >> v1:172.21.15.98:6789/0 conn(0x7f08a80d11e0 legacy=0x7f08a80d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:25.750 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.740+0000 7f08ae7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:50:25.750 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.740+0000 7f08ae7fc640 1 monclient: found mon.noname-b 2024-03-29T11:50:25.750 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.740+0000 7f08ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:50:25.750 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.740+0000 7f08ae7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:50:25.750 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.740+0000 7f08ae7fc640 1 -- 172.21.15.98:0/2992993955 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a80c7a60 con 0x7f08a80324b0 2024-03-29T11:50:25.751 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.740+0000 7f08ae7fc640 10 monclient: _finish_auth 0 2024-03-29T11:50:25.751 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.740+0000 7f08ae7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:50:25.751 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.740+0000 7f08ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:50:25.751 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.740+0000 7f08ae7fc640 1 -- 172.21.15.98:0/2992993955 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f089c001670 con 0x7f08a80324b0 2024-03-29T11:50:25.751 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.741+0000 7f08ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:25.751 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.741+0000 7f08ae7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:50:25.751 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.741+0000 7f08ae7fc640 10 monclient: dump: 2024-03-29T11:50:25.751 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:50:25.751 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:50:25.751 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:25.751 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:25.751 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:25.751 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:25.751 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:50:25.751 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:50:25.752 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:50:25.752 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:25.752 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.741+0000 7f08b49e1640 5 monclient: authenticate success, global_id 4304 2024-03-29T11:50:25.752 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.741+0000 7f08b49e1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:50:25.752 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.741+0000 7f08ae7fc640 1 -- 172.21.15.98:0/2992993955 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f089c0017d0 con 0x7f08a80324b0 2024-03-29T11:50:25.752 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.741+0000 7f08ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:50:25.752 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.741+0000 7f08ae7fc640 1 -- 172.21.15.98:0/2992993955 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f089c001be0 con 0x7f08a80324b0 2024-03-29T11:50:25.752 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.741+0000 7f08b49e1640 10 monclient: get_monmap_and_config success 2024-03-29T11:50:25.752 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.741+0000 7f08b49e1640 4 set_mon_vals no callback set 2024-03-29T11:50:25.752 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.741+0000 7f08ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:25.752 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.741+0000 7f08ae7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:50:25.752 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.741+0000 7f08ae7fc640 10 monclient: dump: 2024-03-29T11:50:25.752 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:50:25.752 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:50:25.752 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:25.753 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:25.753 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:25.753 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:25.753 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:50:25.753 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:50:25.753 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:50:25.753 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:25.753 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.742+0000 7f08b49e1640 10 monclient: shutdown 2024-03-29T11:50:25.753 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.742+0000 7f08b49e1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:50:25.753 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.742+0000 7f08b49e1640 1 -- 172.21.15.98:0/2992993955 >> v1:172.21.15.98:6790/0 conn(0x7f08a80324b0 legacy=0x7f08a80cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:25.753 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.742+0000 7f08b49e1640 1 -- 172.21.15.98:0/2992993955 shutdown_connections 2024-03-29T11:50:25.753 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.742+0000 7f08b49e1640 1 -- 172.21.15.98:0/2992993955 >> 172.21.15.98:0/2992993955 conn(0x7f08a8005c20 msgr2=0x7f08a8006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:50:25.753 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.743+0000 7f08b49e1640 1 -- 172.21.15.98:0/2992993955 shutdown_connections 2024-03-29T11:50:25.753 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.743+0000 7f08b49e1640 1 -- 172.21.15.98:0/2992993955 wait complete. 2024-03-29T11:50:25.754 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.743+0000 7f08b49e1640 10 monclient: build_initial_monmap 2024-03-29T11:50:25.754 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.743+0000 7f08b49e1640 10 monclient: monmap: 2024-03-29T11:50:25.754 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:50:25.754 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:50:25.754 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:50:25.754 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:50:25.754 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:50:25.754 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:25.754 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:50:25.754 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:50:25.754 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:50:25.755 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:25.755 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.744+0000 7f08b49e1640 1 librados: starting msgr at 2024-03-29T11:50:25.755 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.744+0000 7f08b49e1640 1 librados: starting objecter 2024-03-29T11:50:25.755 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.744+0000 7f08b49e1640 1 Processor -- start 2024-03-29T11:50:25.755 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.744+0000 7f08b49e1640 1 -- start start 2024-03-29T11:50:25.756 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.744+0000 7f08b49e1640 1 librados: setting wanted keys 2024-03-29T11:50:25.756 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.744+0000 7f08b49e1640 1 librados: calling monclient init 2024-03-29T11:50:25.756 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.744+0000 7f08b49e1640 10 monclient: init 2024-03-29T11:50:25.756 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.745+0000 7f08b49e1640 10 monclient: _reopen_session rank -1 2024-03-29T11:50:25.756 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.745+0000 7f08b49e1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:50:25.756 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.745+0000 7f08b49e1640 10 monclient(hunting): picked mon.noname-c con 0x7f08a80324b0 addr v1:172.21.15.98:6790/0 2024-03-29T11:50:25.756 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.745+0000 7f08b49e1640 10 monclient(hunting): picked mon.noname-a con 0x7f08a8038c60 addr v1:172.21.15.98:6789/0 2024-03-29T11:50:25.756 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.745+0000 7f08b49e1640 10 monclient(hunting): picked mon.noname-b con 0x7f08a80d11e0 addr v1:172.21.15.121:6789/0 2024-03-29T11:50:25.756 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.745+0000 7f08b49e1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08a80302f0 con 0x7f08a8038c60 2024-03-29T11:50:25.756 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.745+0000 7f08b49e1640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08a802f5f0 con 0x7f08a80d11e0 2024-03-29T11:50:25.756 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.745+0000 7f08b49e1640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08a802f8d0 con 0x7f08a80324b0 2024-03-29T11:50:25.756 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.745+0000 7f08b49e1640 10 monclient(hunting): _renew_subs 2024-03-29T11:50:25.756 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.745+0000 7f08b49e1640 10 monclient(hunting): authenticate will time out at 932.830577s 2024-03-29T11:50:25.762 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.750+0000 7f08aeffd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f08a8038c60 0x7f08a8035d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56137/0 (socket says 172.21.15.98:56137) 2024-03-29T11:50:25.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.750+0000 7f08aeffd640 1 -- 172.21.15.98:0/1397408860 learned_addr learned my addr 172.21.15.98:0/1397408860 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:50:25.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.750+0000 7f08affff640 1 --1- 172.21.15.98:0/1397408860 >> v1:172.21.15.121:6789/0 conn(0x7f08a80d11e0 0x7f08a8163370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:36725/0 (socket says 172.21.15.98:36725) 2024-03-29T11:50:25.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.750+0000 7f089bfff640 10 client.?.objecter ms_handle_connect 0x7f08a8038c60 2024-03-29T11:50:25.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.750+0000 7f089bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:50:25.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.751+0000 7f089bfff640 10 client.?.objecter ms_handle_connect 0x7f08a80d11e0 2024-03-29T11:50:25.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.751+0000 7f089bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:50:25.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.751+0000 7f089bfff640 1 -- 172.21.15.98:0/1397408860 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3772567455 0 0) 0x7f08a80302f0 con 0x7f08a8038c60 2024-03-29T11:50:25.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.751+0000 7f089bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:50:25.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.751+0000 7f089bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:25.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.751+0000 7f089bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:25.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.751+0000 7f089bfff640 10 monclient(hunting): my global_id is 4299 2024-03-29T11:50:25.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.751+0000 7f089bfff640 1 -- 172.21.15.98:0/1397408860 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0894004eb0 con 0x7f08a8038c60 2024-03-29T11:50:25.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.751+0000 7f089bfff640 1 -- 172.21.15.98:0/1397408860 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1787337915 0 0) 0x7f08a802f5f0 con 0x7f08a80d11e0 2024-03-29T11:50:25.764 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.751+0000 7f089bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:50:25.764 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.751+0000 7f089bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:25.764 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.751+0000 7f089bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:25.764 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.751+0000 7f089bfff640 10 monclient(hunting): my global_id is 4324 2024-03-29T11:50:25.764 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.751+0000 7f089bfff640 1 -- 172.21.15.98:0/1397408860 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08a80302f0 con 0x7f08a80d11e0 2024-03-29T11:50:25.764 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.751+0000 7f089bfff640 1 -- 172.21.15.98:0/1397408860 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2347898196 0 0) 0x7f0894004eb0 con 0x7f08a8038c60 2024-03-29T11:50:25.764 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.751+0000 7f089bfff640 1 -- 172.21.15.98:0/1397408860 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08a802f5f0 con 0x7f08a8038c60 2024-03-29T11:50:25.764 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.751+0000 7f089bfff640 1 -- 172.21.15.98:0/1397408860 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f089c001670 con 0x7f08a8038c60 2024-03-29T11:50:25.764 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.752+0000 7f089bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:25.764 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.752+0000 7f089bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:50:25.764 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.752+0000 7f089bfff640 10 monclient(hunting): dump: 2024-03-29T11:50:25.764 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:50:25.764 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:50:25.764 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:25.765 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:25.765 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:25.765 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:25.765 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:50:25.765 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:50:25.765 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:50:25.765 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:25.765 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.752+0000 7f089bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:50:25.765 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.752+0000 7f089bfff640 1 -- 172.21.15.98:0/1397408860 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3620353616 0 0) 0x7f08a802f5f0 con 0x7f08a8038c60 2024-03-29T11:50:25.765 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.752+0000 7f089bfff640 1 -- 172.21.15.98:0/1397408860 >> v1:172.21.15.98:6790/0 conn(0x7f08a80324b0 legacy=0x7f08a802d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:25.765 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.752+0000 7f089bfff640 1 -- 172.21.15.98:0/1397408860 >> v1:172.21.15.121:6789/0 conn(0x7f08a80d11e0 legacy=0x7f08a8163370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:25.765 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.752+0000 7f089bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:50:25.765 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.752+0000 7f089bfff640 1 monclient: found mon.a 2024-03-29T11:50:25.765 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.752+0000 7f089bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:50:25.765 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.752+0000 7f089bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:50:25.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.752+0000 7f089bfff640 1 -- 172.21.15.98:0/1397408860 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a800d4b0 con 0x7f08a8038c60 2024-03-29T11:50:25.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.752+0000 7f089bfff640 10 monclient: _finish_auth 0 2024-03-29T11:50:25.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.752+0000 7f089bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:50:25.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.752+0000 7f089bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:50:25.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.752+0000 7f08b49e1640 5 monclient: authenticate success, global_id 4299 2024-03-29T11:50:25.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.752+0000 7f08b49e1640 10 monclient: _renew_subs 2024-03-29T11:50:25.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.752+0000 7f08b49e1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:50:25.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.752+0000 7f08b49e1640 1 -- 172.21.15.98:0/1397408860 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08a802fd60 con 0x7f08a8038c60 2024-03-29T11:50:25.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.752+0000 7f08b49e1640 10 client.4299.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:50:25.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.752+0000 7f08b49e1640 10 monclient: _renew_subs 2024-03-29T11:50:25.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.752+0000 7f08b49e1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:50:25.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.752+0000 7f08b49e1640 1 -- 172.21.15.98:0/1397408860 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f08a800d150 con 0x7f08a8038c60 2024-03-29T11:50:25.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.752+0000 7f08b49e1640 1 librados: init done 2024-03-29T11:50:25.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.753+0000 7f089bfff640 10 client.4299.objecter ms_handle_connect 0x7f08a80324b0 2024-03-29T11:50:25.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.753+0000 7f089bfff640 10 client.4299.objecter resend_mon_ops 2024-03-29T11:50:25.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.753+0000 7f089bfff640 1 -- 172.21.15.98:0/1397408860 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f089c003a00 con 0x7f08a8038c60 2024-03-29T11:50:25.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.753+0000 7f089bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:50:25.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.753+0000 7f089bfff640 1 -- 172.21.15.98:0/1397408860 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f089c003e50 con 0x7f08a8038c60 2024-03-29T11:50:25.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.753+0000 7f089bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:25.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.753+0000 7f089bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:50:25.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.753+0000 7f089bfff640 10 monclient: dump: 2024-03-29T11:50:25.767 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:50:25.767 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:50:25.767 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:25.767 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:25.767 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:25.767 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:25.767 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:50:25.767 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:50:25.767 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:50:25.767 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:25.768 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.754+0000 7f08b49e1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:25.768 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.754+0000 7f08b49e1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:25.768 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.754+0000 7f08b49e1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:50:25.768 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.754+0000 7f08ad7fa640 4 set_mon_vals no callback set 2024-03-29T11:50:25.768 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.754+0000 7f08b49e1640 1 -- 172.21.15.98:0/1397408860 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08a8007b90 con 0x7f08a8038c60 2024-03-29T11:50:25.768 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.755+0000 7f089bfff640 1 -- 172.21.15.98:0/1397408860 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f089c003fb0 con 0x7f08a8038c60 2024-03-29T11:50:25.768 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.756+0000 7f08af7fe640 10 monclient: get_auth_request con 0x7f089406daf0 auth_method 0 2024-03-29T11:50:25.768 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.756+0000 7f089bfff640 1 -- 172.21.15.98:0/1397408860 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f089c084b70 con 0x7f08a8038c60 2024-03-29T11:50:25.768 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.756+0000 7f089bfff640 10 client.4299.objecter ms_dispatch 0x7f08b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:50:25.768 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.756+0000 7f089bfff640 3 client.4299.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:50:25.768 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.756+0000 7f089bfff640 3 client.4299.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:50:25.768 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.756+0000 7f089bfff640 20 client.4299.objecter dump_active .. 0 homeless 2024-03-29T11:50:25.769 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.757+0000 7f089bfff640 10 client.4299.objecter ms_handle_connect 0x7f089406daf0 2024-03-29T11:50:25.771 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.764+0000 7f089bfff640 1 -- 172.21.15.98:0/1397408860 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f089c050450 con 0x7f08a8038c60 2024-03-29T11:50:25.771 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.764+0000 7f089bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:25.771 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:25.764+0000 7f089bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:50:26.086 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:26.079+0000 7f08b49e1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:26.086 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:26.079+0000 7f08b49e1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:26.086 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:26.079+0000 7f08b49e1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:50:26.086 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:26.079+0000 7f08b49e1640 1 -- 172.21.15.98:0/1397408860 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08a8004cd0 con 0x7f08a8038c60 2024-03-29T11:50:26.088 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:26.082+0000 7f089bfff640 1 -- 172.21.15.98:0/1397408860 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f089c052380 con 0x7f08a8038c60 2024-03-29T11:50:26.089 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:26.082+0000 7f089bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:26.089 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:26.082+0000 7f089bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:50:26.089 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:50:26.089 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:50:26.093 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:26.086+0000 7f08b49e1640 10 librados: watch_flush enter 2024-03-29T11:50:26.093 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:26.086+0000 7f08b49e1640 10 librados: watch_flush exit 2024-03-29T11:50:26.093 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:26.086+0000 7f08b49e1640 20 client.4299.objecter shutdown clearing up homeless session... 2024-03-29T11:50:26.093 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:26.086+0000 7f08b49e1640 10 client.4299.objecter successfully canceled tick 2024-03-29T11:50:26.093 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:26.086+0000 7f08b49e1640 1 -- 172.21.15.98:0/1397408860 >> v1:172.21.15.98:6816/4065186974 conn(0x7f089406daf0 legacy=0x7f089406ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:26.093 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:26.086+0000 7f08b49e1640 10 monclient: shutdown 2024-03-29T11:50:26.093 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:26.086+0000 7f08b49e1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:50:26.094 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:26.086+0000 7f08b49e1640 1 -- 172.21.15.98:0/1397408860 >> v1:172.21.15.98:6789/0 conn(0x7f08a8038c60 legacy=0x7f08a8035d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:26.094 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:26.086+0000 7f08b49e1640 1 -- 172.21.15.98:0/1397408860 shutdown_connections 2024-03-29T11:50:26.094 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:26.087+0000 7f08b49e1640 1 -- 172.21.15.98:0/1397408860 >> 172.21.15.98:0/1397408860 conn(0x7f08a8005c20 msgr2=0x7f08a80043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:50:26.094 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:26.087+0000 7f08b49e1640 1 -- 172.21.15.98:0/1397408860 shutdown_connections 2024-03-29T11:50:26.094 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:26.087+0000 7f08b49e1640 1 -- 172.21.15.98:0/1397408860 wait complete. 2024-03-29T11:50:26.094 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:26.087+0000 7f08b49e1640 1 librados: shutdown 2024-03-29T11:50:26.111 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:50:32.917 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:50:33.062 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.055+0000 7ff803d04640 10 monclient: get_monmap_and_config 2024-03-29T11:50:33.062 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.055+0000 7ff803d04640 10 monclient: build_initial_monmap 2024-03-29T11:50:33.062 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.056+0000 7ff803d04640 10 monclient: monmap: 2024-03-29T11:50:33.062 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:50:33.062 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:50:33.062 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:50:33.056871+0000 2024-03-29T11:50:33.063 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:50:33.056871+0000 2024-03-29T11:50:33.063 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:50:33.063 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:33.063 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:50:33.063 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:50:33.063 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:50:33.063 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:33.063 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.056+0000 7ff803d04640 1 Processor -- start 2024-03-29T11:50:33.064 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.057+0000 7ff803d04640 1 -- start start 2024-03-29T11:50:33.064 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.057+0000 7ff803d04640 10 monclient: init 2024-03-29T11:50:33.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.058+0000 7ff803d04640 10 monclient: _reopen_session rank -1 2024-03-29T11:50:33.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.058+0000 7ff803d04640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:50:33.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.058+0000 7ff803d04640 10 monclient(hunting): picked mon.noname-c con 0x7ff7fc1457f0 addr v1:172.21.15.121:6789/0 2024-03-29T11:50:33.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.058+0000 7ff803d04640 10 monclient(hunting): picked mon.noname-a con 0x7ff7fc13f650 addr v1:172.21.15.98:6789/0 2024-03-29T11:50:33.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.058+0000 7ff803d04640 10 monclient(hunting): picked mon.noname-b con 0x7ff7fc0a9d70 addr v1:172.21.15.98:6790/0 2024-03-29T11:50:33.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.058+0000 7ff803d04640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7fc0aaef0 con 0x7ff7fc13f650 2024-03-29T11:50:33.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.058+0000 7ff803d04640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7fc145bd0 con 0x7ff7fc1457f0 2024-03-29T11:50:33.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.058+0000 7ff803d04640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7fc145eb0 con 0x7ff7fc0a9d70 2024-03-29T11:50:33.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.058+0000 7ff803d04640 10 monclient(hunting): _renew_subs 2024-03-29T11:50:33.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.058+0000 7ff803d04640 10 monclient(hunting): authenticate will time out at 940.143261s 2024-03-29T11:50:33.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.058+0000 7ff801aa0640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7ff7fc1457f0 0x7ff7fc13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:50720/0 (socket says 172.21.15.98:50720) 2024-03-29T11:50:33.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.058+0000 7ff801aa0640 1 -- 172.21.15.98:0/1369328737 learned_addr learned my addr 172.21.15.98:0/1369328737 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:50:33.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.058+0000 7ff800a9e640 1 -- 172.21.15.98:0/1369328737 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2382824862 0 0) 0x7ff7fc145bd0 con 0x7ff7fc1457f0 2024-03-29T11:50:33.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.058+0000 7ff800a9e640 10 monclient(hunting): _negotiate 2024-03-29T11:50:33.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.058+0000 7ff800a9e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:33.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.058+0000 7ff800a9e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:33.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.058+0000 7ff800a9e640 10 monclient(hunting): my global_id is 4327 2024-03-29T11:50:33.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.058+0000 7ff800a9e640 1 -- 172.21.15.98:0/1369328737 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7e8003520 con 0x7ff7fc1457f0 2024-03-29T11:50:33.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.058+0000 7ff800a9e640 1 -- 172.21.15.98:0/1369328737 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1647487258 0 0) 0x7ff7fc145eb0 con 0x7ff7fc0a9d70 2024-03-29T11:50:33.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.058+0000 7ff800a9e640 10 monclient(hunting): _negotiate 2024-03-29T11:50:33.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.058+0000 7ff800a9e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:33.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.058+0000 7ff800a9e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:33.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.058+0000 7ff800a9e640 10 monclient(hunting): my global_id is 4310 2024-03-29T11:50:33.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.058+0000 7ff800a9e640 1 -- 172.21.15.98:0/1369328737 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7fc145bd0 con 0x7ff7fc0a9d70 2024-03-29T11:50:33.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.059+0000 7ff800a9e640 1 -- 172.21.15.98:0/1369328737 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3920321858 0 0) 0x7ff7fc0aaef0 con 0x7ff7fc13f650 2024-03-29T11:50:33.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.059+0000 7ff800a9e640 10 monclient(hunting): _negotiate 2024-03-29T11:50:33.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.059+0000 7ff800a9e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:33.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.059+0000 7ff800a9e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:33.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.059+0000 7ff800a9e640 10 monclient(hunting): my global_id is 4302 2024-03-29T11:50:33.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.059+0000 7ff800a9e640 1 -- 172.21.15.98:0/1369328737 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7fc145eb0 con 0x7ff7fc13f650 2024-03-29T11:50:33.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.059+0000 7ff800a9e640 1 -- 172.21.15.98:0/1369328737 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2310033075 0 0) 0x7ff7e8003520 con 0x7ff7fc1457f0 2024-03-29T11:50:33.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.059+0000 7ff800a9e640 1 -- 172.21.15.98:0/1369328737 >> v1:172.21.15.98:6790/0 conn(0x7ff7fc0a9d70 legacy=0x7ff7fc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:33.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.059+0000 7ff800a9e640 1 -- 172.21.15.98:0/1369328737 >> v1:172.21.15.98:6789/0 conn(0x7ff7fc13f650 legacy=0x7ff7fc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:33.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.059+0000 7ff800a9e640 10 monclient: _finish_hunting 0 2024-03-29T11:50:33.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.059+0000 7ff800a9e640 1 monclient: found mon.noname-c 2024-03-29T11:50:33.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.059+0000 7ff800a9e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:50:33.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.059+0000 7ff800a9e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:50:33.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.059+0000 7ff800a9e640 1 -- 172.21.15.98:0/1369328737 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7fc146340 con 0x7ff7fc1457f0 2024-03-29T11:50:33.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.059+0000 7ff800a9e640 10 monclient: _finish_auth 0 2024-03-29T11:50:33.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.059+0000 7ff800a9e640 10 monclient: _check_auth_tickets 2024-03-29T11:50:33.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.059+0000 7ff800a9e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:50:33.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.059+0000 7ff800a9e640 1 -- 172.21.15.98:0/1369328737 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7ff7f8001a10 con 0x7ff7fc1457f0 2024-03-29T11:50:33.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.059+0000 7ff800a9e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:33.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.059+0000 7ff800a9e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:50:33.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.059+0000 7ff800a9e640 10 monclient: dump: 2024-03-29T11:50:33.071 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:50:33.072 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:50:33.072 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:33.072 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:33.072 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:33.072 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:33.072 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:50:33.072 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:50:33.072 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:50:33.072 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:33.072 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.059+0000 7ff803d04640 5 monclient: authenticate success, global_id 4327 2024-03-29T11:50:33.072 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.059+0000 7ff803d04640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:50:33.072 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.059+0000 7ff800a9e640 1 -- 172.21.15.98:0/1369328737 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff7f8001b70 con 0x7ff7fc1457f0 2024-03-29T11:50:33.072 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.059+0000 7ff800a9e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:50:33.072 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.059+0000 7ff803d04640 10 monclient: get_monmap_and_config success 2024-03-29T11:50:33.072 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.059+0000 7ff803d04640 4 set_mon_vals no callback set 2024-03-29T11:50:33.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.059+0000 7ff800a9e640 1 -- 172.21.15.98:0/1369328737 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7ff7f8001f80 con 0x7ff7fc1457f0 2024-03-29T11:50:33.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.059+0000 7ff800a9e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:33.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.059+0000 7ff800a9e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:50:33.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.059+0000 7ff800a9e640 10 monclient: dump: 2024-03-29T11:50:33.073 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:50:33.073 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:50:33.073 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:33.073 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:33.073 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:33.073 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:33.073 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:50:33.073 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:50:33.073 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:50:33.073 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:33.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.060+0000 7ff803d04640 10 monclient: shutdown 2024-03-29T11:50:33.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.060+0000 7ff803d04640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:50:33.074 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.060+0000 7ff803d04640 1 -- 172.21.15.98:0/1369328737 >> v1:172.21.15.121:6789/0 conn(0x7ff7fc1457f0 legacy=0x7ff7fc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:33.074 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.060+0000 7ff803d04640 1 -- 172.21.15.98:0/1369328737 shutdown_connections 2024-03-29T11:50:33.074 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.060+0000 7ff803d04640 1 -- 172.21.15.98:0/1369328737 >> 172.21.15.98:0/1369328737 conn(0x7ff7fc066a00 msgr2=0x7ff7fc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:50:33.074 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.060+0000 7ff803d04640 1 -- 172.21.15.98:0/1369328737 shutdown_connections 2024-03-29T11:50:33.074 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.060+0000 7ff803d04640 1 -- 172.21.15.98:0/1369328737 wait complete. 2024-03-29T11:50:33.074 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.061+0000 7ff803d04640 10 monclient: build_initial_monmap 2024-03-29T11:50:33.074 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.061+0000 7ff803d04640 10 monclient: monmap: 2024-03-29T11:50:33.074 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:50:33.074 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:50:33.074 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:50:33.074 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:50:33.074 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:50:33.074 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:33.074 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:50:33.075 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:50:33.075 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:50:33.075 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:33.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.062+0000 7ff803d04640 1 librados: starting msgr at 2024-03-29T11:50:33.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.062+0000 7ff803d04640 1 librados: starting objecter 2024-03-29T11:50:33.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.062+0000 7ff803d04640 1 Processor -- start 2024-03-29T11:50:33.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.062+0000 7ff803d04640 1 -- start start 2024-03-29T11:50:33.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.062+0000 7ff803d04640 1 librados: setting wanted keys 2024-03-29T11:50:33.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.062+0000 7ff803d04640 1 librados: calling monclient init 2024-03-29T11:50:33.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.062+0000 7ff803d04640 10 monclient: init 2024-03-29T11:50:33.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.063+0000 7ff803d04640 10 monclient: _reopen_session rank -1 2024-03-29T11:50:33.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.063+0000 7ff803d04640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:50:33.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.063+0000 7ff803d04640 10 monclient(hunting): picked mon.noname-c con 0x7ff7fc0a9d70 addr v1:172.21.15.98:6790/0 2024-03-29T11:50:33.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.063+0000 7ff803d04640 10 monclient(hunting): picked mon.noname-a con 0x7ff7fc13f650 addr v1:172.21.15.98:6789/0 2024-03-29T11:50:33.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.063+0000 7ff803d04640 10 monclient(hunting): picked mon.noname-b con 0x7ff7fc1457f0 addr v1:172.21.15.121:6789/0 2024-03-29T11:50:33.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.063+0000 7ff80129f640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ff7fc13f650 0x7ff7fc1dafa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57368/0 (socket says 172.21.15.98:57368) 2024-03-29T11:50:33.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.063+0000 7ff80129f640 1 -- 172.21.15.98:0/4248089854 learned_addr learned my addr 172.21.15.98:0/4248089854 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:50:33.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.063+0000 7ff803d04640 1 -- 172.21.15.98:0/4248089854 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7fc142100 con 0x7ff7fc13f650 2024-03-29T11:50:33.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.063+0000 7ff803d04640 1 -- 172.21.15.98:0/4248089854 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7fc1423e0 con 0x7ff7fc1457f0 2024-03-29T11:50:33.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.063+0000 7ff803d04640 1 -- 172.21.15.98:0/4248089854 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7fc1daa80 con 0x7ff7fc0a9d70 2024-03-29T11:50:33.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.063+0000 7ff803d04640 10 monclient(hunting): _renew_subs 2024-03-29T11:50:33.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.063+0000 7ff803d04640 10 monclient(hunting): authenticate will time out at 940.148598s 2024-03-29T11:50:33.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.063+0000 7ff7f27fc640 10 client.?.objecter ms_handle_connect 0x7ff7fc13f650 2024-03-29T11:50:33.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.063+0000 7ff7f27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:50:33.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.063+0000 7ff7f27fc640 10 client.?.objecter ms_handle_connect 0x7ff7fc0a9d70 2024-03-29T11:50:33.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.063+0000 7ff7f27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:50:33.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.063+0000 7ff7f27fc640 1 -- 172.21.15.98:0/4248089854 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 420798341 0 0) 0x7ff7fc1daa80 con 0x7ff7fc0a9d70 2024-03-29T11:50:33.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.063+0000 7ff7f27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:50:33.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.063+0000 7ff7f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:33.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.063+0000 7ff7f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:33.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.063+0000 7ff7f27fc640 10 monclient(hunting): my global_id is 4313 2024-03-29T11:50:33.082 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.063+0000 7ff7f27fc640 1 -- 172.21.15.98:0/4248089854 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7e0004eb0 con 0x7ff7fc0a9d70 2024-03-29T11:50:33.082 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.064+0000 7ff7f27fc640 1 -- 172.21.15.98:0/4248089854 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4214905463 0 0) 0x7ff7e0004eb0 con 0x7ff7fc0a9d70 2024-03-29T11:50:33.082 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.064+0000 7ff7f27fc640 1 -- 172.21.15.98:0/4248089854 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7fc1daa80 con 0x7ff7fc0a9d70 2024-03-29T11:50:33.082 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.064+0000 7ff7f27fc640 1 -- 172.21.15.98:0/4248089854 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7ff7f8001e80 con 0x7ff7fc0a9d70 2024-03-29T11:50:33.082 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.064+0000 7ff7f27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:33.082 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.064+0000 7ff7f27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:50:33.082 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.064+0000 7ff7f27fc640 10 monclient(hunting): dump: 2024-03-29T11:50:33.082 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:50:33.082 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:50:33.082 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:33.082 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:33.082 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:33.082 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:33.082 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:50:33.083 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:50:33.083 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:50:33.083 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:33.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.064+0000 7ff7f27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:50:33.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.064+0000 7ff7f27fc640 1 -- 172.21.15.98:0/4248089854 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3457864246 0 0) 0x7ff7fc142100 con 0x7ff7fc13f650 2024-03-29T11:50:33.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.064+0000 7ff7f27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:50:33.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.064+0000 7ff7f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:33.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.064+0000 7ff7f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:33.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.064+0000 7ff7f27fc640 10 monclient(hunting): my global_id is 4305 2024-03-29T11:50:33.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.064+0000 7ff7f27fc640 1 -- 172.21.15.98:0/4248089854 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7e0004eb0 con 0x7ff7fc13f650 2024-03-29T11:50:33.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.064+0000 7ff7f27fc640 1 -- 172.21.15.98:0/4248089854 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1371756738 0 0) 0x7ff7e0004eb0 con 0x7ff7fc13f650 2024-03-29T11:50:33.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.064+0000 7ff7f27fc640 1 -- 172.21.15.98:0/4248089854 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7fc142100 con 0x7ff7fc13f650 2024-03-29T11:50:33.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.064+0000 7ff7f27fc640 1 -- 172.21.15.98:0/4248089854 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 191378871 0 0) 0x7ff7fc1daa80 con 0x7ff7fc0a9d70 2024-03-29T11:50:33.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.065+0000 7ff7f27fc640 1 -- 172.21.15.98:0/4248089854 >> v1:172.21.15.121:6789/0 conn(0x7ff7fc1457f0 legacy=0x7ff7fc1e1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:33.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.065+0000 7ff7f27fc640 1 -- 172.21.15.98:0/4248089854 >> v1:172.21.15.98:6789/0 conn(0x7ff7fc13f650 legacy=0x7ff7fc1dafa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:33.084 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.065+0000 7ff7f27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:50:33.084 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.065+0000 7ff7f27fc640 1 monclient: found mon.c 2024-03-29T11:50:33.084 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.065+0000 7ff7f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:50:33.084 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.065+0000 7ff7f27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:50:33.084 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.065+0000 7ff7f27fc640 1 -- 172.21.15.98:0/4248089854 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7fc1db6b0 con 0x7ff7fc0a9d70 2024-03-29T11:50:33.084 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.065+0000 7ff7f27fc640 10 monclient: _finish_auth 0 2024-03-29T11:50:33.084 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.065+0000 7ff7f27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:50:33.084 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.065+0000 7ff7f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:50:33.084 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.065+0000 7ff803d04640 5 monclient: authenticate success, global_id 4313 2024-03-29T11:50:33.084 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.065+0000 7ff803d04640 10 monclient: _renew_subs 2024-03-29T11:50:33.084 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.065+0000 7ff803d04640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:50:33.084 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.065+0000 7ff803d04640 1 -- 172.21.15.98:0/4248089854 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7fc1db900 con 0x7ff7fc0a9d70 2024-03-29T11:50:33.084 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.065+0000 7ff803d04640 10 client.4313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:50:33.084 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.065+0000 7ff803d04640 10 monclient: _renew_subs 2024-03-29T11:50:33.084 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.065+0000 7ff803d04640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:50:33.085 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.065+0000 7ff803d04640 1 -- 172.21.15.98:0/4248089854 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7fc13e6a0 con 0x7ff7fc0a9d70 2024-03-29T11:50:33.085 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.065+0000 7ff803d04640 1 librados: init done 2024-03-29T11:50:33.085 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.065+0000 7ff7f27fc640 1 -- 172.21.15.98:0/4248089854 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff7f8001fe0 con 0x7ff7fc0a9d70 2024-03-29T11:50:33.085 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.065+0000 7ff7f27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:50:33.085 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.065+0000 7ff7f27fc640 1 -- 172.21.15.98:0/4248089854 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7ff7f8004fb0 con 0x7ff7fc0a9d70 2024-03-29T11:50:33.085 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.065+0000 7ff7f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:33.085 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.065+0000 7ff7f27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:50:33.085 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.065+0000 7ff7f27fc640 10 monclient: dump: 2024-03-29T11:50:33.085 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:50:33.085 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:50:33.085 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:33.085 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:33.085 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:33.085 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:33.085 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:50:33.086 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:50:33.086 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:50:33.086 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:33.086 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.065+0000 7ff7f37fe640 4 set_mon_vals no callback set 2024-03-29T11:50:33.086 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.067+0000 7ff803d04640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:33.086 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.067+0000 7ff803d04640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:33.086 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.067+0000 7ff803d04640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:50:33.086 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.067+0000 7ff803d04640 1 -- 172.21.15.98:0/4248089854 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7e4005080 con 0x7ff7fc0a9d70 2024-03-29T11:50:33.086 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.067+0000 7ff7f27fc640 1 -- 172.21.15.98:0/4248089854 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7ff7f801af80 con 0x7ff7fc0a9d70 2024-03-29T11:50:33.086 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.068+0000 7ff80129f640 10 monclient: get_auth_request con 0x7ff7e006e050 auth_method 0 2024-03-29T11:50:33.086 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.068+0000 7ff7f27fc640 1 -- 172.21.15.98:0/4248089854 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7ff7f8084da0 con 0x7ff7fc0a9d70 2024-03-29T11:50:33.086 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.068+0000 7ff7f27fc640 10 client.4313.objecter ms_dispatch 0x7ff7fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:50:33.086 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.068+0000 7ff7f27fc640 3 client.4313.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:50:33.086 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.068+0000 7ff7f27fc640 3 client.4313.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:50:33.086 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.068+0000 7ff7f27fc640 20 client.4313.objecter dump_active .. 0 homeless 2024-03-29T11:50:33.087 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.068+0000 7ff7f27fc640 10 client.4313.objecter ms_handle_connect 0x7ff7e006e050 2024-03-29T11:50:33.087 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.072+0000 7ff7f27fc640 1 -- 172.21.15.98:0/4248089854 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff7f80504b0 con 0x7ff7fc0a9d70 2024-03-29T11:50:33.087 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.072+0000 7ff7f27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:33.087 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.072+0000 7ff7f27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:50:33.420 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.413+0000 7ff803d04640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:33.420 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.413+0000 7ff803d04640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:33.420 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.413+0000 7ff803d04640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:50:33.420 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.413+0000 7ff803d04640 1 -- 172.21.15.98:0/4248089854 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7e4005330 con 0x7ff7fc0a9d70 2024-03-29T11:50:33.421 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.415+0000 7ff7f27fc640 1 -- 172.21.15.98:0/4248089854 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7ff7f80525b0 con 0x7ff7fc0a9d70 2024-03-29T11:50:33.421 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.415+0000 7ff7f27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:33.421 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:50:33.421 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:50:33.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.415+0000 7ff7f27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:50:33.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.418+0000 7ff803d04640 10 librados: watch_flush enter 2024-03-29T11:50:33.425 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.418+0000 7ff803d04640 10 librados: watch_flush exit 2024-03-29T11:50:33.425 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.418+0000 7ff803d04640 20 client.4313.objecter shutdown clearing up homeless session... 2024-03-29T11:50:33.426 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.418+0000 7ff803d04640 10 client.4313.objecter successfully canceled tick 2024-03-29T11:50:33.426 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.418+0000 7ff803d04640 1 -- 172.21.15.98:0/4248089854 >> v1:172.21.15.98:6816/4065186974 conn(0x7ff7e006e050 legacy=0x7ff7e0070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:33.426 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.418+0000 7ff803d04640 10 monclient: shutdown 2024-03-29T11:50:33.426 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.418+0000 7ff803d04640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:50:33.426 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.418+0000 7ff803d04640 1 -- 172.21.15.98:0/4248089854 >> v1:172.21.15.98:6790/0 conn(0x7ff7fc0a9d70 legacy=0x7ff7fc142800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:33.426 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.419+0000 7ff803d04640 1 -- 172.21.15.98:0/4248089854 shutdown_connections 2024-03-29T11:50:33.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.419+0000 7ff803d04640 1 -- 172.21.15.98:0/4248089854 >> 172.21.15.98:0/4248089854 conn(0x7ff7fc066a00 msgr2=0x7ff7fc0b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:50:33.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.419+0000 7ff803d04640 1 -- 172.21.15.98:0/4248089854 shutdown_connections 2024-03-29T11:50:33.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.419+0000 7ff803d04640 1 -- 172.21.15.98:0/4248089854 wait complete. 2024-03-29T11:50:33.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:33.419+0000 7ff803d04640 1 librados: shutdown 2024-03-29T11:50:33.440 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:50:40.246 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:50:40.399 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.392+0000 7fe202f20640 10 monclient: get_monmap_and_config 2024-03-29T11:50:40.399 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.392+0000 7fe202f20640 10 monclient: build_initial_monmap 2024-03-29T11:50:40.399 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.392+0000 7fe202f20640 10 monclient: monmap: 2024-03-29T11:50:40.399 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:50:40.399 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:50:40.399 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:50:40.393626+0000 2024-03-29T11:50:40.400 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:50:40.393626+0000 2024-03-29T11:50:40.400 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:50:40.400 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:40.400 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:50:40.400 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:50:40.400 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:50:40.400 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:40.400 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.393+0000 7fe202f20640 1 Processor -- start 2024-03-29T11:50:40.401 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.394+0000 7fe202f20640 1 -- start start 2024-03-29T11:50:40.401 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.394+0000 7fe202f20640 10 monclient: init 2024-03-29T11:50:40.401 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.395+0000 7fe202f20640 10 monclient: _reopen_session rank -1 2024-03-29T11:50:40.401 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.395+0000 7fe202f20640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:50:40.401 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.395+0000 7fe202f20640 10 monclient(hunting): picked mon.noname-c con 0x7fe1fc0b0860 addr v1:172.21.15.121:6789/0 2024-03-29T11:50:40.401 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.395+0000 7fe202f20640 10 monclient(hunting): picked mon.noname-b con 0x7fe1fc072450 addr v1:172.21.15.98:6790/0 2024-03-29T11:50:40.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.395+0000 7fe202f20640 10 monclient(hunting): picked mon.noname-a con 0x7fe1fc13ee40 addr v1:172.21.15.98:6789/0 2024-03-29T11:50:40.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.395+0000 7fe202f20640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1fc0ad840 con 0x7fe1fc13ee40 2024-03-29T11:50:40.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.395+0000 7fe202f20640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1fc13c830 con 0x7fe1fc0b0860 2024-03-29T11:50:40.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.395+0000 7fe200cbc640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fe1fc0b0860 0x7fe1fc0acfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:48378/0 (socket says 172.21.15.98:48378) 2024-03-29T11:50:40.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.395+0000 7fe200cbc640 1 -- 172.21.15.98:0/3277827763 learned_addr learned my addr 172.21.15.98:0/3277827763 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:50:40.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.395+0000 7fe202f20640 1 -- 172.21.15.98:0/3277827763 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1fc13c9c0 con 0x7fe1fc072450 2024-03-29T11:50:40.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.395+0000 7fe202f20640 10 monclient(hunting): _renew_subs 2024-03-29T11:50:40.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.395+0000 7fe202f20640 10 monclient(hunting): authenticate will time out at 947.480598s 2024-03-29T11:50:40.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.399+0000 7fe1fb7fe640 1 -- 172.21.15.98:0/3277827763 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 214116452 0 0) 0x7fe1fc13c830 con 0x7fe1fc0b0860 2024-03-29T11:50:40.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.399+0000 7fe1fb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:50:40.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.399+0000 7fe1fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:40.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.399+0000 7fe1fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:40.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.399+0000 7fe1fb7fe640 10 monclient(hunting): my global_id is 4330 2024-03-29T11:50:40.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.399+0000 7fe1fb7fe640 1 -- 172.21.15.98:0/3277827763 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1e4003520 con 0x7fe1fc0b0860 2024-03-29T11:50:40.409 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.400+0000 7fe1fb7fe640 1 -- 172.21.15.98:0/3277827763 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2190851392 0 0) 0x7fe1fc0ad840 con 0x7fe1fc13ee40 2024-03-29T11:50:40.409 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.400+0000 7fe1fb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:50:40.409 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.400+0000 7fe1fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:40.409 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.400+0000 7fe1fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:40.409 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.400+0000 7fe1fb7fe640 10 monclient(hunting): my global_id is 4308 2024-03-29T11:50:40.409 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.400+0000 7fe1fb7fe640 1 -- 172.21.15.98:0/3277827763 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1fc13c830 con 0x7fe1fc13ee40 2024-03-29T11:50:40.409 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.400+0000 7fe1fb7fe640 1 -- 172.21.15.98:0/3277827763 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1863874436 0 0) 0x7fe1fc13c9c0 con 0x7fe1fc072450 2024-03-29T11:50:40.409 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.400+0000 7fe1fb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:50:40.409 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.400+0000 7fe1fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:40.409 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.400+0000 7fe1fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:40.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.400+0000 7fe1fb7fe640 10 monclient(hunting): my global_id is 4316 2024-03-29T11:50:40.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.400+0000 7fe1fb7fe640 1 -- 172.21.15.98:0/3277827763 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1fc0ad840 con 0x7fe1fc072450 2024-03-29T11:50:40.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.400+0000 7fe1fb7fe640 1 -- 172.21.15.98:0/3277827763 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3638678693 0 0) 0x7fe1fc0ad840 con 0x7fe1fc072450 2024-03-29T11:50:40.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.400+0000 7fe1fb7fe640 1 -- 172.21.15.98:0/3277827763 >> v1:172.21.15.121:6789/0 conn(0x7fe1fc0b0860 legacy=0x7fe1fc0acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:40.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.400+0000 7fe1fb7fe640 1 -- 172.21.15.98:0/3277827763 >> v1:172.21.15.98:6789/0 conn(0x7fe1fc13ee40 legacy=0x7fe1fc13f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:40.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.400+0000 7fe1fb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:50:40.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.400+0000 7fe1fb7fe640 1 monclient: found mon.noname-b 2024-03-29T11:50:40.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.401+0000 7fe1fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:50:40.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.401+0000 7fe1fb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:50:40.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.401+0000 7fe1fb7fe640 1 -- 172.21.15.98:0/3277827763 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1fc13f970 con 0x7fe1fc072450 2024-03-29T11:50:40.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.401+0000 7fe1fb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:50:40.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.401+0000 7fe1fb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:50:40.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.401+0000 7fe1fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:50:40.411 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.401+0000 7fe202f20640 5 monclient: authenticate success, global_id 4316 2024-03-29T11:50:40.411 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.401+0000 7fe202f20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:50:40.411 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.401+0000 7fe1fb7fe640 1 -- 172.21.15.98:0/3277827763 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fe1ec0014e0 con 0x7fe1fc072450 2024-03-29T11:50:40.411 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.401+0000 7fe1fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:40.411 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.401+0000 7fe1fb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:50:40.411 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.401+0000 7fe1fb7fe640 10 monclient: dump: 2024-03-29T11:50:40.411 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:50:40.411 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:50:40.411 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:40.411 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:40.411 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:40.411 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:40.411 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:50:40.411 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:50:40.411 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:50:40.412 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:40.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.401+0000 7fe202f20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:50:40.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.401+0000 7fe1fb7fe640 1 -- 172.21.15.98:0/3277827763 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1ec001700 con 0x7fe1fc072450 2024-03-29T11:50:40.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.401+0000 7fe1fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:50:40.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.401+0000 7fe202f20640 10 monclient: get_monmap_and_config success 2024-03-29T11:50:40.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.401+0000 7fe202f20640 4 set_mon_vals no callback set 2024-03-29T11:50:40.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.402+0000 7fe202f20640 10 monclient: shutdown 2024-03-29T11:50:40.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.402+0000 7fe202f20640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:50:40.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.402+0000 7fe202f20640 1 -- 172.21.15.98:0/3277827763 >> v1:172.21.15.98:6790/0 conn(0x7fe1fc072450 legacy=0x7fe1fc13c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:40.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.402+0000 7fe202f20640 1 -- 172.21.15.98:0/3277827763 shutdown_connections 2024-03-29T11:50:40.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.402+0000 7fe202f20640 1 -- 172.21.15.98:0/3277827763 >> 172.21.15.98:0/3277827763 conn(0x7fe1fc06b660 msgr2=0x7fe1fc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:50:40.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.403+0000 7fe202f20640 1 -- 172.21.15.98:0/3277827763 shutdown_connections 2024-03-29T11:50:40.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.403+0000 7fe202f20640 1 -- 172.21.15.98:0/3277827763 wait complete. 2024-03-29T11:50:40.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.404+0000 7fe202f20640 10 monclient: build_initial_monmap 2024-03-29T11:50:40.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.404+0000 7fe202f20640 10 monclient: monmap: 2024-03-29T11:50:40.414 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:50:40.414 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:50:40.414 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:50:40.414 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:50:40.415 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:50:40.415 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:40.415 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:50:40.415 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:50:40.415 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:50:40.415 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:40.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.405+0000 7fe202f20640 1 librados: starting msgr at 2024-03-29T11:50:40.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.405+0000 7fe202f20640 1 librados: starting objecter 2024-03-29T11:50:40.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.405+0000 7fe202f20640 1 Processor -- start 2024-03-29T11:50:40.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.405+0000 7fe202f20640 1 -- start start 2024-03-29T11:50:40.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.405+0000 7fe202f20640 1 librados: setting wanted keys 2024-03-29T11:50:40.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.405+0000 7fe202f20640 1 librados: calling monclient init 2024-03-29T11:50:40.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.405+0000 7fe202f20640 10 monclient: init 2024-03-29T11:50:40.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.407+0000 7fe202f20640 10 monclient: _reopen_session rank -1 2024-03-29T11:50:40.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.407+0000 7fe202f20640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:50:40.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.407+0000 7fe202f20640 10 monclient(hunting): picked mon.noname-c con 0x7fe1fc072450 addr v1:172.21.15.98:6790/0 2024-03-29T11:50:40.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.407+0000 7fe202f20640 10 monclient(hunting): picked mon.noname-b con 0x7fe1fc0b0860 addr v1:172.21.15.121:6789/0 2024-03-29T11:50:40.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.409+0000 7fe1fbfff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fe1fc0b0860 0x7fe1fc1357b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:48692/0 (socket says 172.21.15.98:48692) 2024-03-29T11:50:40.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.409+0000 7fe200cbc640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fe1fc072450 0x7fe1fc0b3360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:35522/0 (socket says 172.21.15.98:35522) 2024-03-29T11:50:40.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.409+0000 7fe202f20640 10 monclient(hunting): picked mon.noname-a con 0x7fe1fc13ee40 addr v1:172.21.15.98:6789/0 2024-03-29T11:50:40.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.409+0000 7fe202f20640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1fc0b3c50 con 0x7fe1fc13ee40 2024-03-29T11:50:40.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.410+0000 7fe202f20640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1fc0b2ef0 con 0x7fe1fc0b0860 2024-03-29T11:50:40.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.410+0000 7fe202f20640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1fc0b3080 con 0x7fe1fc072450 2024-03-29T11:50:40.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.410+0000 7fe202f20640 10 monclient(hunting): _renew_subs 2024-03-29T11:50:40.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.410+0000 7fe202f20640 10 monclient(hunting): authenticate will time out at 947.495093s 2024-03-29T11:50:40.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.410+0000 7fe1fbfff640 1 -- 172.21.15.98:0/1598334816 learned_addr learned my addr 172.21.15.98:0/1598334816 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:50:40.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.410+0000 7fe1f97fa640 10 client.?.objecter ms_handle_connect 0x7fe1fc072450 2024-03-29T11:50:40.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.410+0000 7fe1f97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:50:40.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.410+0000 7fe1f97fa640 10 client.?.objecter ms_handle_connect 0x7fe1fc0b0860 2024-03-29T11:50:40.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.410+0000 7fe1f97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:50:40.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.410+0000 7fe1f97fa640 1 -- 172.21.15.98:0/1598334816 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1386770150 0 0) 0x7fe1fc0b3080 con 0x7fe1fc072450 2024-03-29T11:50:40.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.410+0000 7fe1f97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:50:40.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.410+0000 7fe1f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:40.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.410+0000 7fe1f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:40.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.410+0000 7fe1f97fa640 10 monclient(hunting): my global_id is 4319 2024-03-29T11:50:40.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.410+0000 7fe1f97fa640 1 -- 172.21.15.98:0/1598334816 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1e8004eb0 con 0x7fe1fc072450 2024-03-29T11:50:40.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.410+0000 7fe1f97fa640 1 -- 172.21.15.98:0/1598334816 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2137446706 0 0) 0x7fe1fc0b2ef0 con 0x7fe1fc0b0860 2024-03-29T11:50:40.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.410+0000 7fe1f97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:50:40.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.410+0000 7fe1f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:40.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.411+0000 7fe1f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:40.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.411+0000 7fe1f97fa640 10 monclient(hunting): my global_id is 4333 2024-03-29T11:50:40.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.411+0000 7fe1f97fa640 1 -- 172.21.15.98:0/1598334816 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1fc0b3080 con 0x7fe1fc0b0860 2024-03-29T11:50:40.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.411+0000 7fe1f97fa640 10 client.?.objecter ms_handle_connect 0x7fe1fc13ee40 2024-03-29T11:50:40.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.411+0000 7fe1f97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:50:40.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.411+0000 7fe1f97fa640 1 -- 172.21.15.98:0/1598334816 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 150470300 0 0) 0x7fe1e8004eb0 con 0x7fe1fc072450 2024-03-29T11:50:40.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.411+0000 7fe1f97fa640 1 -- 172.21.15.98:0/1598334816 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1fc0b2ef0 con 0x7fe1fc072450 2024-03-29T11:50:40.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.411+0000 7fe1f97fa640 1 -- 172.21.15.98:0/1598334816 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2276290769 0 0) 0x7fe1fc0b3080 con 0x7fe1fc0b0860 2024-03-29T11:50:40.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.411+0000 7fe1f97fa640 1 -- 172.21.15.98:0/1598334816 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1e8004eb0 con 0x7fe1fc0b0860 2024-03-29T11:50:40.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.411+0000 7fe1f97fa640 1 -- 172.21.15.98:0/1598334816 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 802015009 0 0) 0x7fe1fc0b3c50 con 0x7fe1fc13ee40 2024-03-29T11:50:40.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.412+0000 7fe1f97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:50:40.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.412+0000 7fe1f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:40.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.412+0000 7fe1f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:40.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.412+0000 7fe1f97fa640 10 monclient(hunting): my global_id is 4311 2024-03-29T11:50:40.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.412+0000 7fe1f97fa640 1 -- 172.21.15.98:0/1598334816 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1fc0b3080 con 0x7fe1fc13ee40 2024-03-29T11:50:40.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.412+0000 7fe1f97fa640 1 -- 172.21.15.98:0/1598334816 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fe1f0001950 con 0x7fe1fc072450 2024-03-29T11:50:40.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.412+0000 7fe1f97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:40.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.412+0000 7fe1f97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:50:40.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.412+0000 7fe1f97fa640 10 monclient(hunting): dump: 2024-03-29T11:50:40.424 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:50:40.424 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:50:40.425 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:40.425 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:40.425 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:40.425 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:40.425 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:50:40.425 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:50:40.425 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:50:40.425 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:40.425 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.412+0000 7fe1f97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:50:40.425 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.412+0000 7fe1f97fa640 1 -- 172.21.15.98:0/1598334816 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fe1ec003790 con 0x7fe1fc0b0860 2024-03-29T11:50:40.425 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.412+0000 7fe1f97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:40.425 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.412+0000 7fe1f97fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:50:40.425 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.412+0000 7fe1f97fa640 10 monclient(hunting): dump: 2024-03-29T11:50:40.425 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:50:40.425 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:50:40.425 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:40.426 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:40.426 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:40.426 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:40.426 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:50:40.426 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:50:40.426 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:50:40.426 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:40.426 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.412+0000 7fe1f97fa640 1 -- 172.21.15.98:0/1598334816 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 367047010 0 0) 0x7fe1fc0b2ef0 con 0x7fe1fc072450 2024-03-29T11:50:40.426 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.412+0000 7fe1f97fa640 1 -- 172.21.15.98:0/1598334816 >> v1:172.21.15.121:6789/0 conn(0x7fe1fc0b0860 legacy=0x7fe1fc1357b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:40.426 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.413+0000 7fe1f97fa640 1 -- 172.21.15.98:0/1598334816 >> v1:172.21.15.98:6789/0 conn(0x7fe1fc13ee40 legacy=0x7fe1fc1d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:40.426 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.413+0000 7fe1f97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:50:40.426 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.413+0000 7fe1f97fa640 1 monclient: found mon.c 2024-03-29T11:50:40.426 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.413+0000 7fe1f97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:50:40.426 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.413+0000 7fe1f97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:50:40.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.413+0000 7fe1f97fa640 1 -- 172.21.15.98:0/1598334816 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1fc135ec0 con 0x7fe1fc072450 2024-03-29T11:50:40.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.413+0000 7fe1f97fa640 10 monclient: _finish_auth 0 2024-03-29T11:50:40.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.413+0000 7fe1f97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:50:40.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.413+0000 7fe1f97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:50:40.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.413+0000 7fe202f20640 5 monclient: authenticate success, global_id 4319 2024-03-29T11:50:40.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.413+0000 7fe202f20640 10 monclient: _renew_subs 2024-03-29T11:50:40.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.413+0000 7fe202f20640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:50:40.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.413+0000 7fe202f20640 1 -- 172.21.15.98:0/1598334816 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1fc136170 con 0x7fe1fc072450 2024-03-29T11:50:40.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.413+0000 7fe202f20640 10 client.4319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:50:40.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.413+0000 7fe202f20640 10 monclient: _renew_subs 2024-03-29T11:50:40.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.413+0000 7fe202f20640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:50:40.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.413+0000 7fe202f20640 1 -- 172.21.15.98:0/1598334816 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1fc142760 con 0x7fe1fc072450 2024-03-29T11:50:40.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.413+0000 7fe202f20640 1 librados: init done 2024-03-29T11:50:40.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.413+0000 7fe1f97fa640 1 -- 172.21.15.98:0/1598334816 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1f00047e0 con 0x7fe1fc072450 2024-03-29T11:50:40.428 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.413+0000 7fe1f97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:50:40.428 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.413+0000 7fe1f97fa640 1 -- 172.21.15.98:0/1598334816 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fe1f0004af0 con 0x7fe1fc072450 2024-03-29T11:50:40.428 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.413+0000 7fe1fa7fc640 4 set_mon_vals no callback set 2024-03-29T11:50:40.428 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.414+0000 7fe1f97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:40.428 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.414+0000 7fe1f97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:50:40.428 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.414+0000 7fe1f97fa640 10 monclient: dump: 2024-03-29T11:50:40.428 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:50:40.428 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:50:40.428 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:40.428 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:40.428 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:40.428 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:40.428 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:50:40.428 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:50:40.429 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:50:40.429 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:40.429 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.416+0000 7fe1f97fa640 1 -- 172.21.15.98:0/1598334816 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7fe1f001aac0 con 0x7fe1fc072450 2024-03-29T11:50:40.430 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.416+0000 7fe1f97fa640 1 -- 172.21.15.98:0/1598334816 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7fe1f00849a0 con 0x7fe1fc072450 2024-03-29T11:50:40.430 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.417+0000 7fe1f97fa640 10 client.4319.objecter ms_dispatch 0x7fe1fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:50:40.430 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.417+0000 7fe1f97fa640 3 client.4319.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:50:40.430 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.417+0000 7fe1f97fa640 3 client.4319.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:50:40.430 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.417+0000 7fe1f97fa640 20 client.4319.objecter dump_active .. 0 homeless 2024-03-29T11:50:40.430 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.417+0000 7fe1fbfff640 10 monclient: get_auth_request con 0x7fe1e806df70 auth_method 0 2024-03-29T11:50:40.430 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.417+0000 7fe202f20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:40.430 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.417+0000 7fe202f20640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:40.431 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.417+0000 7fe202f20640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:50:40.431 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.417+0000 7fe202f20640 1 -- 172.21.15.98:0/1598334816 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1dc005080 con 0x7fe1fc072450 2024-03-29T11:50:40.431 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.417+0000 7fe1f97fa640 10 client.4319.objecter ms_handle_connect 0x7fe1e806df70 2024-03-29T11:50:40.432 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.426+0000 7fe1f97fa640 1 -- 172.21.15.98:0/1598334816 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe1f0050280 con 0x7fe1fc072450 2024-03-29T11:50:40.433 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.426+0000 7fe1f97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:40.433 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.426+0000 7fe1f97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:50:40.756 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.749+0000 7fe202f20640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:40.756 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.749+0000 7fe202f20640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:40.756 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.749+0000 7fe202f20640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:50:40.756 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.749+0000 7fe202f20640 1 -- 172.21.15.98:0/1598334816 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1dc005330 con 0x7fe1fc072450 2024-03-29T11:50:40.757 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.750+0000 7fe1f97fa640 1 -- 172.21.15.98:0/1598334816 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7fe1f00521b0 con 0x7fe1fc072450 2024-03-29T11:50:40.757 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.750+0000 7fe1f97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:40.757 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.750+0000 7fe1f97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:50:40.757 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:50:40.757 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:50:40.761 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.754+0000 7fe202f20640 10 librados: watch_flush enter 2024-03-29T11:50:40.761 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.754+0000 7fe202f20640 10 librados: watch_flush exit 2024-03-29T11:50:40.761 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.754+0000 7fe202f20640 20 client.4319.objecter shutdown clearing up homeless session... 2024-03-29T11:50:40.761 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.754+0000 7fe202f20640 10 client.4319.objecter successfully canceled tick 2024-03-29T11:50:40.761 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.754+0000 7fe202f20640 1 -- 172.21.15.98:0/1598334816 >> v1:172.21.15.98:6816/4065186974 conn(0x7fe1e806df70 legacy=0x7fe1e8070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:40.761 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.754+0000 7fe202f20640 10 monclient: shutdown 2024-03-29T11:50:40.762 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.754+0000 7fe202f20640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:50:40.762 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.754+0000 7fe202f20640 1 -- 172.21.15.98:0/1598334816 >> v1:172.21.15.98:6790/0 conn(0x7fe1fc072450 legacy=0x7fe1fc0b3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:40.762 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.756+0000 7fe202f20640 1 -- 172.21.15.98:0/1598334816 shutdown_connections 2024-03-29T11:50:40.762 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.756+0000 7fe202f20640 1 -- 172.21.15.98:0/1598334816 >> 172.21.15.98:0/1598334816 conn(0x7fe1fc06b660 msgr2=0x7fe1fc06ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:50:40.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.756+0000 7fe202f20640 1 -- 172.21.15.98:0/1598334816 shutdown_connections 2024-03-29T11:50:40.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.756+0000 7fe202f20640 1 -- 172.21.15.98:0/1598334816 wait complete. 2024-03-29T11:50:40.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:40.756+0000 7fe202f20640 1 librados: shutdown 2024-03-29T11:50:40.779 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:50:47.585 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:50:47.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.723+0000 7f749655d640 10 monclient: get_monmap_and_config 2024-03-29T11:50:47.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.723+0000 7f749655d640 10 monclient: build_initial_monmap 2024-03-29T11:50:47.731 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.723+0000 7f749655d640 10 monclient: monmap: 2024-03-29T11:50:47.731 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:50:47.731 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:50:47.731 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:50:47.724415+0000 2024-03-29T11:50:47.731 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:50:47.724415+0000 2024-03-29T11:50:47.731 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:50:47.731 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:47.731 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:50:47.731 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:50:47.731 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:50:47.731 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:47.732 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.725+0000 7f749655d640 1 Processor -- start 2024-03-29T11:50:47.732 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.725+0000 7f749655d640 1 -- start start 2024-03-29T11:50:47.732 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.725+0000 7f749655d640 10 monclient: init 2024-03-29T11:50:47.733 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.726+0000 7f749655d640 10 monclient: _reopen_session rank -1 2024-03-29T11:50:47.734 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.726+0000 7f749655d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:50:47.734 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.727+0000 7f749655d640 10 monclient(hunting): picked mon.noname-a con 0x7f74901457f0 addr v1:172.21.15.98:6789/0 2024-03-29T11:50:47.734 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.727+0000 7f749655d640 10 monclient(hunting): picked mon.noname-b con 0x7f749013f650 addr v1:172.21.15.98:6790/0 2024-03-29T11:50:47.734 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.727+0000 7f749655d640 10 monclient(hunting): picked mon.noname-c con 0x7f74900a9d70 addr v1:172.21.15.121:6789/0 2024-03-29T11:50:47.734 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.727+0000 7f749655d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74900aaef0 con 0x7f74901457f0 2024-03-29T11:50:47.734 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.727+0000 7f7494afa640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f74900a9d70 0x7f749014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:60862/0 (socket says 172.21.15.98:60862) 2024-03-29T11:50:47.734 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.727+0000 7f7494afa640 1 -- 172.21.15.98:0/2981819184 learned_addr learned my addr 172.21.15.98:0/2981819184 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:50:47.734 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.727+0000 7f749655d640 1 -- 172.21.15.98:0/2981819184 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7490145bd0 con 0x7f74900a9d70 2024-03-29T11:50:47.734 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.727+0000 7f749655d640 1 -- 172.21.15.98:0/2981819184 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7490145eb0 con 0x7f749013f650 2024-03-29T11:50:47.734 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.727+0000 7f749655d640 10 monclient(hunting): _renew_subs 2024-03-29T11:50:47.734 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.727+0000 7f749655d640 10 monclient(hunting): authenticate will time out at 954.812749s 2024-03-29T11:50:47.735 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.728+0000 7f748effd640 1 -- 172.21.15.98:0/2981819184 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4016331535 0 0) 0x7f7490145bd0 con 0x7f74900a9d70 2024-03-29T11:50:47.735 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.728+0000 7f748effd640 10 monclient(hunting): _negotiate 2024-03-29T11:50:47.735 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.728+0000 7f748effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:47.735 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.728+0000 7f748effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:47.735 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.728+0000 7f748effd640 10 monclient(hunting): my global_id is 4336 2024-03-29T11:50:47.735 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.728+0000 7f748effd640 1 -- 172.21.15.98:0/2981819184 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7478003520 con 0x7f74900a9d70 2024-03-29T11:50:47.736 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.729+0000 7f748effd640 1 -- 172.21.15.98:0/2981819184 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1744842087 0 0) 0x7f7478003520 con 0x7f74900a9d70 2024-03-29T11:50:47.736 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.729+0000 7f748effd640 1 -- 172.21.15.98:0/2981819184 >> v1:172.21.15.98:6790/0 conn(0x7f749013f650 legacy=0x7f74900b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:50:47.740 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.731+0000 7f748effd640 1 -- 172.21.15.98:0/2981819184 >> v1:172.21.15.98:6789/0 conn(0x7f74901457f0 legacy=0x7f749013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:47.740 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.732+0000 7f748effd640 10 monclient: _finish_hunting 0 2024-03-29T11:50:47.740 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.732+0000 7f748effd640 1 monclient: found mon.noname-c 2024-03-29T11:50:47.740 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.732+0000 7f748effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:50:47.740 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.732+0000 7f748effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:50:47.740 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.732+0000 7f748effd640 1 -- 172.21.15.98:0/2981819184 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7490146340 con 0x7f74900a9d70 2024-03-29T11:50:47.740 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.732+0000 7f748effd640 10 monclient: _finish_auth 0 2024-03-29T11:50:47.740 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.732+0000 7f748effd640 10 monclient: _check_auth_tickets 2024-03-29T11:50:47.740 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.732+0000 7f748effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:50:47.740 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.732+0000 7f749655d640 5 monclient: authenticate success, global_id 4336 2024-03-29T11:50:47.741 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.732+0000 7f749655d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:50:47.741 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.732+0000 7f748effd640 1 -- 172.21.15.98:0/2981819184 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f7488001a10 con 0x7f74900a9d70 2024-03-29T11:50:47.741 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.732+0000 7f748effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:47.741 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.732+0000 7f748effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:50:47.741 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.732+0000 7f748effd640 10 monclient: dump: 2024-03-29T11:50:47.741 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:50:47.741 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:50:47.741 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:47.741 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:47.741 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:47.741 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:47.741 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:50:47.741 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:50:47.742 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:50:47.742 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:47.742 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.733+0000 7f749655d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:50:47.742 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.733+0000 7f748effd640 1 -- 172.21.15.98:0/2981819184 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7488001ca0 con 0x7f74900a9d70 2024-03-29T11:50:47.742 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.733+0000 7f748effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:50:47.742 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.733+0000 7f749655d640 10 monclient: get_monmap_and_config success 2024-03-29T11:50:47.743 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.733+0000 7f749655d640 4 set_mon_vals no callback set 2024-03-29T11:50:47.743 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.733+0000 7f748effd640 1 -- 172.21.15.98:0/2981819184 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f7488002070 con 0x7f74900a9d70 2024-03-29T11:50:47.743 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.733+0000 7f748effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:47.743 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.733+0000 7f748effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:50:47.743 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.733+0000 7f748effd640 10 monclient: dump: 2024-03-29T11:50:47.743 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:50:47.743 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:50:47.744 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:47.744 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:47.744 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:47.744 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:47.744 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:50:47.744 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:50:47.745 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:50:47.745 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:47.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.734+0000 7f749655d640 10 monclient: shutdown 2024-03-29T11:50:47.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.734+0000 7f749655d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:50:47.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.734+0000 7f749655d640 1 -- 172.21.15.98:0/2981819184 >> v1:172.21.15.121:6789/0 conn(0x7f74900a9d70 legacy=0x7f749014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:47.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.734+0000 7f749655d640 1 -- 172.21.15.98:0/2981819184 shutdown_connections 2024-03-29T11:50:47.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.734+0000 7f749655d640 1 -- 172.21.15.98:0/2981819184 >> 172.21.15.98:0/2981819184 conn(0x7f7490066a00 msgr2=0x7f7490066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:50:47.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.735+0000 7f749655d640 1 -- 172.21.15.98:0/2981819184 shutdown_connections 2024-03-29T11:50:47.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.735+0000 7f749655d640 1 -- 172.21.15.98:0/2981819184 wait complete. 2024-03-29T11:50:47.761 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.736+0000 7f749655d640 10 monclient: build_initial_monmap 2024-03-29T11:50:47.761 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.736+0000 7f749655d640 10 monclient: monmap: 2024-03-29T11:50:47.761 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:50:47.761 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:50:47.761 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:50:47.761 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:50:47.762 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:50:47.762 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:47.762 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:50:47.762 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:50:47.762 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:50:47.762 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:47.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.736+0000 7f749655d640 1 librados: starting msgr at 2024-03-29T11:50:47.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.736+0000 7f749655d640 1 librados: starting objecter 2024-03-29T11:50:47.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.737+0000 7f749655d640 1 Processor -- start 2024-03-29T11:50:47.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.737+0000 7f749655d640 1 -- start start 2024-03-29T11:50:47.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.737+0000 7f749655d640 1 librados: setting wanted keys 2024-03-29T11:50:47.764 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.737+0000 7f749655d640 1 librados: calling monclient init 2024-03-29T11:50:47.764 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.737+0000 7f749655d640 10 monclient: init 2024-03-29T11:50:47.764 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.738+0000 7f749655d640 10 monclient: _reopen_session rank -1 2024-03-29T11:50:47.764 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.738+0000 7f749655d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:50:47.764 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.741+0000 7f749655d640 10 monclient(hunting): picked mon.noname-a con 0x7f74900a9d70 addr v1:172.21.15.98:6789/0 2024-03-29T11:50:47.764 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.741+0000 7f749655d640 10 monclient(hunting): picked mon.noname-c con 0x7f749013f650 addr v1:172.21.15.98:6790/0 2024-03-29T11:50:47.764 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.741+0000 7f749655d640 10 monclient(hunting): picked mon.noname-b con 0x7f74901457f0 addr v1:172.21.15.121:6789/0 2024-03-29T11:50:47.765 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.741+0000 7f749655d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74900adb10 con 0x7f74900a9d70 2024-03-29T11:50:47.765 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.741+0000 7f749655d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74900af1a0 con 0x7f74901457f0 2024-03-29T11:50:47.765 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.741+0000 7f749655d640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74900adfa0 con 0x7f749013f650 2024-03-29T11:50:47.765 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.741+0000 7f749655d640 10 monclient(hunting): _renew_subs 2024-03-29T11:50:47.765 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.741+0000 7f749655d640 10 monclient(hunting): authenticate will time out at 954.826835s 2024-03-29T11:50:47.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.741+0000 7f748ffff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f74900a9d70 0x7f74900ad290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:42169/0 (socket says 172.21.15.98:42169) 2024-03-29T11:50:47.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.741+0000 7f748ffff640 1 -- 172.21.15.98:0/2644045819 learned_addr learned my addr 172.21.15.98:0/2644045819 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:50:47.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.742+0000 7f748cff9640 10 client.?.objecter ms_handle_connect 0x7f74900a9d70 2024-03-29T11:50:47.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.742+0000 7f748cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:50:47.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.742+0000 7f748cff9640 1 -- 172.21.15.98:0/2644045819 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2273771668 0 0) 0x7f74900adb10 con 0x7f74900a9d70 2024-03-29T11:50:47.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.742+0000 7f748cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:50:47.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.742+0000 7f748cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:47.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.742+0000 7f748cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:47.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.742+0000 7f748cff9640 10 monclient(hunting): my global_id is 4317 2024-03-29T11:50:47.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.742+0000 7f748cff9640 1 -- 172.21.15.98:0/2644045819 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f747c004eb0 con 0x7f74900a9d70 2024-03-29T11:50:47.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.742+0000 7f748cff9640 10 client.?.objecter ms_handle_connect 0x7f74901457f0 2024-03-29T11:50:47.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.742+0000 7f748cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:50:47.768 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.742+0000 7f748cff9640 1 -- 172.21.15.98:0/2644045819 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2745922189 0 0) 0x7f74900af1a0 con 0x7f74901457f0 2024-03-29T11:50:47.768 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.742+0000 7f748cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:50:47.768 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.742+0000 7f748cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:47.768 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.742+0000 7f748cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:47.768 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.742+0000 7f748cff9640 10 monclient(hunting): my global_id is 4339 2024-03-29T11:50:47.768 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.742+0000 7f748cff9640 1 -- 172.21.15.98:0/2644045819 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74900adb10 con 0x7f74901457f0 2024-03-29T11:50:47.769 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.742+0000 7f748cff9640 1 -- 172.21.15.98:0/2644045819 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3640681594 0 0) 0x7f747c004eb0 con 0x7f74900a9d70 2024-03-29T11:50:47.769 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.742+0000 7f748cff9640 1 -- 172.21.15.98:0/2644045819 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74900af1a0 con 0x7f74900a9d70 2024-03-29T11:50:47.769 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.742+0000 7f748cff9640 1 -- 172.21.15.98:0/2644045819 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f7484001870 con 0x7f74900a9d70 2024-03-29T11:50:47.769 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.742+0000 7f748cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:47.769 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.742+0000 7f748cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:50:47.769 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.742+0000 7f748cff9640 10 monclient(hunting): dump: 2024-03-29T11:50:47.770 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:50:47.770 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:50:47.770 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:47.770 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:47.770 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:47.770 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:47.771 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:50:47.771 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:50:47.771 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:50:47.771 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:47.771 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.742+0000 7f748cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:50:47.771 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.743+0000 7f748cff9640 1 -- 172.21.15.98:0/2644045819 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4150215335 0 0) 0x7f74900af1a0 con 0x7f74900a9d70 2024-03-29T11:50:47.772 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.743+0000 7f748cff9640 1 -- 172.21.15.98:0/2644045819 >> v1:172.21.15.98:6790/0 conn(0x7f749013f650 legacy=0x7f74901da440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:50:47.772 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.745+0000 7f748cff9640 1 -- 172.21.15.98:0/2644045819 >> v1:172.21.15.121:6789/0 conn(0x7f74901457f0 legacy=0x7f74901e19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:47.772 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.745+0000 7f748cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:50:47.772 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.745+0000 7f748cff9640 1 monclient: found mon.a 2024-03-29T11:50:47.772 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.745+0000 7f748cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:50:47.772 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.745+0000 7f748cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:50:47.772 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.745+0000 7f748cff9640 1 -- 172.21.15.98:0/2644045819 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74900acf90 con 0x7f74900a9d70 2024-03-29T11:50:47.772 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.745+0000 7f748cff9640 10 monclient: _finish_auth 0 2024-03-29T11:50:47.773 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.745+0000 7f748cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:50:47.773 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.745+0000 7f748cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:50:47.773 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.745+0000 7f749655d640 5 monclient: authenticate success, global_id 4317 2024-03-29T11:50:47.773 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.745+0000 7f749655d640 10 monclient: _renew_subs 2024-03-29T11:50:47.773 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.745+0000 7f749655d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:50:47.773 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.745+0000 7f749655d640 1 -- 172.21.15.98:0/2644045819 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7490141dd0 con 0x7f74900a9d70 2024-03-29T11:50:47.773 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.745+0000 7f749655d640 10 client.4317.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:50:47.773 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.745+0000 7f749655d640 10 monclient: _renew_subs 2024-03-29T11:50:47.773 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.745+0000 7f749655d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:50:47.774 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.745+0000 7f749655d640 1 -- 172.21.15.98:0/2644045819 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7490142170 con 0x7f74900a9d70 2024-03-29T11:50:47.774 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.745+0000 7f749655d640 1 librados: init done 2024-03-29T11:50:47.774 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.746+0000 7f748cff9640 1 -- 172.21.15.98:0/2644045819 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7484003610 con 0x7f74900a9d70 2024-03-29T11:50:47.774 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.746+0000 7f748cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:50:47.774 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.746+0000 7f748cff9640 1 -- 172.21.15.98:0/2644045819 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f7484001d80 con 0x7f74900a9d70 2024-03-29T11:50:47.774 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.746+0000 7f748cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:47.774 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.746+0000 7f748cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:50:47.774 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.746+0000 7f748cff9640 10 monclient: dump: 2024-03-29T11:50:47.775 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:50:47.775 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:50:47.775 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:47.775 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:47.775 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:47.775 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:47.775 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:50:47.776 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:50:47.776 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:50:47.776 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:47.776 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.747+0000 7f748e7fc640 4 set_mon_vals no callback set 2024-03-29T11:50:47.776 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.747+0000 7f749655d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:47.776 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.748+0000 7f749655d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:47.776 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.748+0000 7f749655d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:50:47.776 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.748+0000 7f749655d640 1 -- 172.21.15.98:0/2644045819 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7470005080 con 0x7f74900a9d70 2024-03-29T11:50:47.776 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.751+0000 7f748cff9640 1 -- 172.21.15.98:0/2644045819 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f7484003ae0 con 0x7f74900a9d70 2024-03-29T11:50:47.776 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.752+0000 7f748cff9640 1 -- 172.21.15.98:0/2644045819 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f7484084950 con 0x7f74900a9d70 2024-03-29T11:50:47.777 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.752+0000 7f748cff9640 10 client.4317.objecter ms_dispatch 0x7f7490000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:50:47.778 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.752+0000 7f748cff9640 3 client.4317.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:50:47.778 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.752+0000 7f748cff9640 3 client.4317.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:50:47.778 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.752+0000 7f748cff9640 20 client.4317.objecter dump_active .. 0 homeless 2024-03-29T11:50:47.778 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.753+0000 7f748f7fe640 10 monclient: get_auth_request con 0x7f747c06daa0 auth_method 0 2024-03-29T11:50:47.778 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.755+0000 7f748cff9640 10 client.4317.objecter ms_handle_connect 0x7f747c06daa0 2024-03-29T11:50:47.778 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.760+0000 7f748cff9640 1 -- 172.21.15.98:0/2644045819 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7484050230 con 0x7f74900a9d70 2024-03-29T11:50:47.778 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.760+0000 7f748cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:47.778 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:47.760+0000 7f748cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:50:48.087 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:48.080+0000 7f749655d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:48.087 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:48.080+0000 7f749655d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:48.087 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:48.080+0000 7f749655d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:50:48.087 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:48.080+0000 7f749655d640 1 -- 172.21.15.98:0/2644045819 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7470005330 con 0x7f74900a9d70 2024-03-29T11:50:48.088 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:50:48.088 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:50:48.089 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:48.081+0000 7f748cff9640 1 -- 172.21.15.98:0/2644045819 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f7484052160 con 0x7f74900a9d70 2024-03-29T11:50:48.089 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:48.081+0000 7f748cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:48.089 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:48.081+0000 7f748cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:50:48.092 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:48.084+0000 7f749655d640 10 librados: watch_flush enter 2024-03-29T11:50:48.092 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:48.084+0000 7f749655d640 10 librados: watch_flush exit 2024-03-29T11:50:48.092 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:48.084+0000 7f749655d640 20 client.4317.objecter shutdown clearing up homeless session... 2024-03-29T11:50:48.092 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:48.084+0000 7f749655d640 10 client.4317.objecter successfully canceled tick 2024-03-29T11:50:48.092 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:48.085+0000 7f749655d640 1 -- 172.21.15.98:0/2644045819 >> v1:172.21.15.98:6816/4065186974 conn(0x7f747c06daa0 legacy=0x7f747c06ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:48.092 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:48.085+0000 7f749655d640 10 monclient: shutdown 2024-03-29T11:50:48.092 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:48.085+0000 7f749655d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:50:48.092 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:48.085+0000 7f749655d640 1 -- 172.21.15.98:0/2644045819 >> v1:172.21.15.98:6789/0 conn(0x7f74900a9d70 legacy=0x7f74900ad290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:48.092 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:48.085+0000 7f749655d640 1 -- 172.21.15.98:0/2644045819 shutdown_connections 2024-03-29T11:50:48.092 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:48.085+0000 7f749655d640 1 -- 172.21.15.98:0/2644045819 >> 172.21.15.98:0/2644045819 conn(0x7f7490066a00 msgr2=0x7f74900b43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:50:48.092 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:48.085+0000 7f749655d640 1 -- 172.21.15.98:0/2644045819 shutdown_connections 2024-03-29T11:50:48.092 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:48.085+0000 7f749655d640 1 -- 172.21.15.98:0/2644045819 wait complete. 2024-03-29T11:50:48.092 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:48.085+0000 7f749655d640 1 librados: shutdown 2024-03-29T11:50:48.109 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:50:54.916 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:50:55.058 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.051+0000 7fe06a657640 10 monclient: get_monmap_and_config 2024-03-29T11:50:55.058 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.051+0000 7fe06a657640 10 monclient: build_initial_monmap 2024-03-29T11:50:55.058 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.051+0000 7fe06a657640 10 monclient: monmap: 2024-03-29T11:50:55.058 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:50:55.059 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:50:55.059 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:50:55.052758+0000 2024-03-29T11:50:55.059 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:50:55.052758+0000 2024-03-29T11:50:55.059 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:50:55.059 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:55.059 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:50:55.059 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:50:55.059 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:50:55.059 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:55.060 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.053+0000 7fe06a657640 1 Processor -- start 2024-03-29T11:50:55.060 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.053+0000 7fe06a657640 1 -- start start 2024-03-29T11:50:55.060 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.053+0000 7fe06a657640 10 monclient: init 2024-03-29T11:50:55.061 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.054+0000 7fe06a657640 10 monclient: _reopen_session rank -1 2024-03-29T11:50:55.061 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.054+0000 7fe06a657640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:50:55.061 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.054+0000 7fe06a657640 10 monclient(hunting): picked mon.noname-a con 0x7fe0640b4890 addr v1:172.21.15.98:6789/0 2024-03-29T11:50:55.061 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.054+0000 7fe06a657640 10 monclient(hunting): picked mon.noname-c con 0x7fe06413c040 addr v1:172.21.15.121:6789/0 2024-03-29T11:50:55.061 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.054+0000 7fe06a657640 10 monclient(hunting): picked mon.noname-b con 0x7fe0641427e0 addr v1:172.21.15.98:6790/0 2024-03-29T11:50:55.061 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.054+0000 7fe06a657640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0640b42f0 con 0x7fe0640b4890 2024-03-29T11:50:55.062 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.054+0000 7fe06a657640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0640aa7a0 con 0x7fe06413c040 2024-03-29T11:50:55.062 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.054+0000 7fe06a657640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0640aaa10 con 0x7fe0641427e0 2024-03-29T11:50:55.062 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.054+0000 7fe06a657640 10 monclient(hunting): _renew_subs 2024-03-29T11:50:55.062 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.054+0000 7fe06a657640 10 monclient(hunting): authenticate will time out at 962.139558s 2024-03-29T11:50:55.063 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.057+0000 7fe068bf4640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fe0641427e0 0x7fe06414d880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:33997/0 (socket says 172.21.15.98:33997) 2024-03-29T11:50:55.063 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.057+0000 7fe068bf4640 1 -- 172.21.15.98:0/3150282780 learned_addr learned my addr 172.21.15.98:0/3150282780 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:50:55.067 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.058+0000 7fe062ffd640 1 -- 172.21.15.98:0/3150282780 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4205251296 0 0) 0x7fe0640aaa10 con 0x7fe0641427e0 2024-03-29T11:50:55.067 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.058+0000 7fe062ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:50:55.067 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.058+0000 7fe062ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:55.067 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.058+0000 7fe062ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:55.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.058+0000 7fe062ffd640 10 monclient(hunting): my global_id is 4322 2024-03-29T11:50:55.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.058+0000 7fe062ffd640 1 -- 172.21.15.98:0/3150282780 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe04c003520 con 0x7fe0641427e0 2024-03-29T11:50:55.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.058+0000 7fe062ffd640 1 -- 172.21.15.98:0/3150282780 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 254607288 0 0) 0x7fe04c003520 con 0x7fe0641427e0 2024-03-29T11:50:55.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.058+0000 7fe062ffd640 1 -- 172.21.15.98:0/3150282780 >> v1:172.21.15.121:6789/0 conn(0x7fe06413c040 legacy=0x7fe0641420d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:55.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.058+0000 7fe062ffd640 1 -- 172.21.15.98:0/3150282780 >> v1:172.21.15.98:6789/0 conn(0x7fe0640b4890 legacy=0x7fe06413b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:55.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.058+0000 7fe062ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:50:55.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.058+0000 7fe062ffd640 1 monclient: found mon.noname-b 2024-03-29T11:50:55.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.058+0000 7fe062ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:50:55.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.058+0000 7fe062ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:50:55.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.058+0000 7fe062ffd640 1 -- 172.21.15.98:0/3150282780 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0640aacf0 con 0x7fe0641427e0 2024-03-29T11:50:55.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.058+0000 7fe062ffd640 10 monclient: _finish_auth 0 2024-03-29T11:50:55.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.058+0000 7fe062ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:50:55.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.059+0000 7fe062ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:50:55.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.059+0000 7fe062ffd640 1 -- 172.21.15.98:0/3150282780 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fe054001690 con 0x7fe0641427e0 2024-03-29T11:50:55.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.059+0000 7fe06a657640 5 monclient: authenticate success, global_id 4322 2024-03-29T11:50:55.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.059+0000 7fe06a657640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:50:55.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.059+0000 7fe062ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:55.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.059+0000 7fe062ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:50:55.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.059+0000 7fe062ffd640 10 monclient: dump: 2024-03-29T11:50:55.069 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:50:55.069 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:50:55.069 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:55.069 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:55.069 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:55.069 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:55.069 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:50:55.069 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:50:55.069 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:50:55.069 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:55.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.059+0000 7fe062ffd640 1 -- 172.21.15.98:0/3150282780 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0540018b0 con 0x7fe0641427e0 2024-03-29T11:50:55.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.059+0000 7fe06a657640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:50:55.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.059+0000 7fe062ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:50:55.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.059+0000 7fe062ffd640 1 -- 172.21.15.98:0/3150282780 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fe054001c80 con 0x7fe0641427e0 2024-03-29T11:50:55.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.059+0000 7fe06a657640 10 monclient: get_monmap_and_config success 2024-03-29T11:50:55.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.059+0000 7fe06a657640 4 set_mon_vals no callback set 2024-03-29T11:50:55.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.060+0000 7fe06a657640 10 monclient: shutdown 2024-03-29T11:50:55.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.060+0000 7fe06a657640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:50:55.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.060+0000 7fe06a657640 1 -- 172.21.15.98:0/3150282780 >> v1:172.21.15.98:6790/0 conn(0x7fe0641427e0 legacy=0x7fe06414d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:55.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.060+0000 7fe06a657640 1 -- 172.21.15.98:0/3150282780 shutdown_connections 2024-03-29T11:50:55.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.061+0000 7fe06a657640 1 -- 172.21.15.98:0/3150282780 >> 172.21.15.98:0/3150282780 conn(0x7fe06406d180 msgr2=0x7fe064066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:50:55.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.061+0000 7fe062ffd640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:50:55.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.062+0000 7fe06a657640 1 -- 172.21.15.98:0/3150282780 shutdown_connections 2024-03-29T11:50:55.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.063+0000 7fe06a657640 1 -- 172.21.15.98:0/3150282780 wait complete. 2024-03-29T11:50:55.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.064+0000 7fe06a657640 10 monclient: build_initial_monmap 2024-03-29T11:50:55.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.064+0000 7fe06a657640 10 monclient: monmap: 2024-03-29T11:50:55.071 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:50:55.071 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:50:55.071 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:50:55.071 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:50:55.071 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:50:55.072 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:55.072 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:50:55.072 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:50:55.072 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:50:55.072 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:55.072 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.064+0000 7fe06a657640 1 librados: starting msgr at 2024-03-29T11:50:55.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.067+0000 7fe06a657640 1 librados: starting objecter 2024-03-29T11:50:55.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.067+0000 7fe06a657640 1 Processor -- start 2024-03-29T11:50:55.074 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.067+0000 7fe06a657640 1 -- start start 2024-03-29T11:50:55.074 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.067+0000 7fe06a657640 1 librados: setting wanted keys 2024-03-29T11:50:55.074 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.067+0000 7fe06a657640 1 librados: calling monclient init 2024-03-29T11:50:55.074 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.067+0000 7fe06a657640 10 monclient: init 2024-03-29T11:50:55.074 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.068+0000 7fe06a657640 10 monclient: _reopen_session rank -1 2024-03-29T11:50:55.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.068+0000 7fe06a657640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:50:55.076 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.069+0000 7fe06a657640 10 monclient(hunting): picked mon.noname-c con 0x7fe0640b4890 addr v1:172.21.15.98:6790/0 2024-03-29T11:50:55.076 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.069+0000 7fe06a657640 10 monclient(hunting): picked mon.noname-a con 0x7fe06413c040 addr v1:172.21.15.98:6789/0 2024-03-29T11:50:55.076 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.069+0000 7fe06a657640 10 monclient(hunting): picked mon.noname-b con 0x7fe0640ab4b0 addr v1:172.21.15.121:6789/0 2024-03-29T11:50:55.076 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.069+0000 7fe06a657640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0640ac510 con 0x7fe06413c040 2024-03-29T11:50:55.077 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.069+0000 7fe06a657640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0641da230 con 0x7fe0640ab4b0 2024-03-29T11:50:55.077 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.069+0000 7fe06a657640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe06413e700 con 0x7fe0640b4890 2024-03-29T11:50:55.077 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.069+0000 7fe06a657640 10 monclient(hunting): _renew_subs 2024-03-29T11:50:55.077 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.069+0000 7fe06a657640 10 monclient(hunting): authenticate will time out at 962.154504s 2024-03-29T11:50:55.077 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.069+0000 7fe068bf4640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fe0640ab4b0 0x7fe0641e2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:58504/0 (socket says 172.21.15.98:58504) 2024-03-29T11:50:55.077 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.069+0000 7fe068bf4640 1 -- 172.21.15.98:0/3598552166 learned_addr learned my addr 172.21.15.98:0/3598552166 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:50:55.077 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.070+0000 7fe060ff9640 10 client.?.objecter ms_handle_connect 0x7fe0640ab4b0 2024-03-29T11:50:55.077 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.070+0000 7fe060ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:50:55.077 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.070+0000 7fe060ff9640 1 -- 172.21.15.98:0/3598552166 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2356030436 0 0) 0x7fe0641da230 con 0x7fe0640ab4b0 2024-03-29T11:50:55.077 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.070+0000 7fe060ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:50:55.077 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.070+0000 7fe060ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:50:55.077 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.070+0000 7fe060ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:50:55.077 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.070+0000 7fe060ff9640 10 monclient(hunting): my global_id is 4345 2024-03-29T11:50:55.077 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.070+0000 7fe060ff9640 1 -- 172.21.15.98:0/3598552166 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe050004eb0 con 0x7fe0640ab4b0 2024-03-29T11:50:55.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.071+0000 7fe060ff9640 1 -- 172.21.15.98:0/3598552166 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 605738969 0 0) 0x7fe050004eb0 con 0x7fe0640ab4b0 2024-03-29T11:50:55.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.071+0000 7fe060ff9640 1 -- 172.21.15.98:0/3598552166 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0641da230 con 0x7fe0640ab4b0 2024-03-29T11:50:55.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.071+0000 7fe060ff9640 1 -- 172.21.15.98:0/3598552166 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fe054001550 con 0x7fe0640ab4b0 2024-03-29T11:50:55.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.071+0000 7fe060ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:55.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.071+0000 7fe060ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:50:55.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.071+0000 7fe060ff9640 10 monclient(hunting): dump: 2024-03-29T11:50:55.080 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:50:55.080 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:50:55.080 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:55.080 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:55.080 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:55.080 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:55.080 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:50:55.080 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:50:55.080 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:50:55.080 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:55.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.071+0000 7fe060ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:50:55.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.071+0000 7fe060ff9640 1 -- 172.21.15.98:0/3598552166 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 825260769 0 0) 0x7fe0641da230 con 0x7fe0640ab4b0 2024-03-29T11:50:55.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.071+0000 7fe060ff9640 1 -- 172.21.15.98:0/3598552166 >> v1:172.21.15.98:6790/0 conn(0x7fe0640b4890 legacy=0x7fe0641da750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:50:55.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.072+0000 7fe060ff9640 1 -- 172.21.15.98:0/3598552166 >> v1:172.21.15.98:6789/0 conn(0x7fe06413c040 legacy=0x7fe06413ae00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:50:55.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.072+0000 7fe060ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:50:55.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.072+0000 7fe060ff9640 1 monclient: found mon.b 2024-03-29T11:50:55.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.072+0000 7fe060ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:50:55.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.072+0000 7fe060ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:50:55.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.072+0000 7fe060ff9640 1 -- 172.21.15.98:0/3598552166 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe06413f800 con 0x7fe0640ab4b0 2024-03-29T11:50:55.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.072+0000 7fe060ff9640 10 monclient: _finish_auth 0 2024-03-29T11:50:55.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.072+0000 7fe060ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:50:55.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.072+0000 7fe060ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:50:55.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.072+0000 7fe06a657640 5 monclient: authenticate success, global_id 4345 2024-03-29T11:50:55.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.072+0000 7fe06a657640 10 monclient: _renew_subs 2024-03-29T11:50:55.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.072+0000 7fe06a657640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:50:55.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.072+0000 7fe06a657640 1 -- 172.21.15.98:0/3598552166 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe06413c590 con 0x7fe0640ab4b0 2024-03-29T11:50:55.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.072+0000 7fe06a657640 10 client.4345.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:50:55.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.072+0000 7fe06a657640 10 monclient: _renew_subs 2024-03-29T11:50:55.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.072+0000 7fe06a657640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:50:55.082 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.072+0000 7fe06a657640 1 -- 172.21.15.98:0/3598552166 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0641db040 con 0x7fe0640ab4b0 2024-03-29T11:50:55.082 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.072+0000 7fe06a657640 1 librados: init done 2024-03-29T11:50:55.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.073+0000 7fe06a657640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:55.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.073+0000 7fe06a657640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:55.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.073+0000 7fe060ff9640 1 -- 172.21.15.98:0/3598552166 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0540016b0 con 0x7fe0640ab4b0 2024-03-29T11:50:55.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.073+0000 7fe06a657640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:50:55.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.073+0000 7fe06a657640 1 -- 172.21.15.98:0/3598552166 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0641da420 con 0x7fe0640ab4b0 2024-03-29T11:50:55.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.073+0000 7fe060ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:50:55.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.073+0000 7fe0627fc640 4 set_mon_vals no callback set 2024-03-29T11:50:55.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.074+0000 7fe060ff9640 1 -- 172.21.15.98:0/3598552166 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fe054004c30 con 0x7fe0640ab4b0 2024-03-29T11:50:55.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.075+0000 7fe060ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:50:55.084 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.075+0000 7fe060ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:50:55.084 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.075+0000 7fe060ff9640 10 monclient: dump: 2024-03-29T11:50:55.084 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:50:55.084 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:50:55.084 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:55.084 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:50:55.084 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:50:55.084 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:50:55.084 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:50:55.084 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:50:55.084 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:50:55.084 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:50:55.085 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.076+0000 7fe060ff9640 1 -- 172.21.15.98:0/3598552166 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7fe05401ac00 con 0x7fe0640ab4b0 2024-03-29T11:50:55.085 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.077+0000 7fe063fff640 10 monclient: get_auth_request con 0x7fe05006dc70 auth_method 0 2024-03-29T11:50:55.085 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.077+0000 7fe060ff9640 1 -- 172.21.15.98:0/3598552166 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7fe054084a90 con 0x7fe0640ab4b0 2024-03-29T11:50:55.085 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.077+0000 7fe060ff9640 10 client.4345.objecter ms_dispatch 0x7fe064000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:50:55.085 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.077+0000 7fe060ff9640 3 client.4345.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:50:55.085 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.077+0000 7fe060ff9640 3 client.4345.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:50:55.085 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.078+0000 7fe060ff9640 20 client.4345.objecter dump_active .. 0 homeless 2024-03-29T11:50:55.085 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.078+0000 7fe060ff9640 10 client.4345.objecter ms_handle_connect 0x7fe05006dc70 2024-03-29T11:50:55.091 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.085+0000 7fe060ff9640 1 -- 172.21.15.98:0/3598552166 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe0540501a0 con 0x7fe0640ab4b0 2024-03-29T11:50:55.091 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.085+0000 7fe060ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:50:55.091 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.085+0000 7fe060ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:50:55.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.408+0000 7fe06a657640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:55.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.408+0000 7fe06a657640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:55.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.408+0000 7fe06a657640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:50:55.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.408+0000 7fe06a657640 1 -- 172.21.15.98:0/3598552166 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0640b1760 con 0x7fe0640ab4b0 2024-03-29T11:50:55.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.411+0000 7fe060ff9640 1 -- 172.21.15.98:0/3598552166 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7fe0540522a0 con 0x7fe0640ab4b0 2024-03-29T11:50:55.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.411+0000 7fe060ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:50:55.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.411+0000 7fe060ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:50:55.418 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:50:55.419 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:50:55.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.415+0000 7fe06a657640 10 librados: watch_flush enter 2024-03-29T11:50:55.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.415+0000 7fe06a657640 10 librados: watch_flush exit 2024-03-29T11:50:55.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.415+0000 7fe06a657640 20 client.4345.objecter shutdown clearing up homeless session... 2024-03-29T11:50:55.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.415+0000 7fe06a657640 10 client.4345.objecter successfully canceled tick 2024-03-29T11:50:55.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.415+0000 7fe06a657640 1 -- 172.21.15.98:0/3598552166 >> v1:172.21.15.98:6816/4065186974 conn(0x7fe05006dc70 legacy=0x7fe050070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:55.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.415+0000 7fe06a657640 10 monclient: shutdown 2024-03-29T11:50:55.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.415+0000 7fe06a657640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:50:55.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.415+0000 7fe06a657640 1 -- 172.21.15.98:0/3598552166 >> v1:172.21.15.121:6789/0 conn(0x7fe0640ab4b0 legacy=0x7fe0641e2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:50:55.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.415+0000 7fe06a657640 1 -- 172.21.15.98:0/3598552166 shutdown_connections 2024-03-29T11:50:55.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.415+0000 7fe06a657640 1 -- 172.21.15.98:0/3598552166 >> 172.21.15.98:0/3598552166 conn(0x7fe06406d180 msgr2=0x7fe0640b51e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:50:55.425 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.418+0000 7fe06a657640 1 -- 172.21.15.98:0/3598552166 shutdown_connections 2024-03-29T11:50:55.425 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.418+0000 7fe06a657640 1 -- 172.21.15.98:0/3598552166 wait complete. 2024-03-29T11:50:55.425 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:50:55.418+0000 7fe06a657640 1 librados: shutdown 2024-03-29T11:50:55.443 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:51:02.250 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:51:02.401 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.393+0000 7f830288b640 10 monclient: get_monmap_and_config 2024-03-29T11:51:02.401 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.393+0000 7f830288b640 10 monclient: build_initial_monmap 2024-03-29T11:51:02.401 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.393+0000 7f830288b640 10 monclient: monmap: 2024-03-29T11:51:02.401 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:51:02.401 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:02.401 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:51:02.394129+0000 2024-03-29T11:51:02.401 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:51:02.394129+0000 2024-03-29T11:51:02.401 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:51:02.401 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:02.401 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:51:02.401 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:51:02.401 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:51:02.402 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:02.402 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.394+0000 7f830288b640 1 Processor -- start 2024-03-29T11:51:02.402 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.394+0000 7f830288b640 1 -- start start 2024-03-29T11:51:02.402 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.394+0000 7f830288b640 10 monclient: init 2024-03-29T11:51:02.403 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.395+0000 7f830288b640 10 monclient: _reopen_session rank -1 2024-03-29T11:51:02.403 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.395+0000 7f830288b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:51:02.403 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.395+0000 7f830288b640 10 monclient(hunting): picked mon.noname-c con 0x7f82fc071f40 addr v1:172.21.15.121:6789/0 2024-03-29T11:51:02.403 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.395+0000 7f830288b640 10 monclient(hunting): picked mon.noname-a con 0x7f82fc0adcb0 addr v1:172.21.15.98:6789/0 2024-03-29T11:51:02.403 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.395+0000 7f830288b640 10 monclient(hunting): picked mon.noname-b con 0x7f82fc13f9a0 addr v1:172.21.15.98:6790/0 2024-03-29T11:51:02.403 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.395+0000 7f830288b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82fc0af1b0 con 0x7f82fc0adcb0 2024-03-29T11:51:02.403 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.395+0000 7f830288b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82fc0b0030 con 0x7f82fc071f40 2024-03-29T11:51:02.403 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.395+0000 7f830288b640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82fc0aec40 con 0x7f82fc13f9a0 2024-03-29T11:51:02.403 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.395+0000 7f830288b640 10 monclient(hunting): _renew_subs 2024-03-29T11:51:02.403 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.395+0000 7f830288b640 10 monclient(hunting): authenticate will time out at 969.480927s 2024-03-29T11:51:02.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.398+0000 7f8301889640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f82fc071f40 0x7f82fc13f290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:43290/0 (socket says 172.21.15.98:43290) 2024-03-29T11:51:02.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.398+0000 7f8301889640 1 -- 172.21.15.98:0/3146449179 learned_addr learned my addr 172.21.15.98:0/3146449179 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:51:02.409 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.399+0000 7f8300887640 1 -- 172.21.15.98:0/3146449179 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1675018048 0 0) 0x7f82fc0aec40 con 0x7f82fc13f9a0 2024-03-29T11:51:02.409 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.399+0000 7f8300887640 10 monclient(hunting): _negotiate 2024-03-29T11:51:02.409 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.399+0000 7f8300887640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:02.409 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.399+0000 7f8300887640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:02.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.399+0000 7f8300887640 10 monclient(hunting): my global_id is 4325 2024-03-29T11:51:02.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.399+0000 7f8300887640 1 -- 172.21.15.98:0/3146449179 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82e8003520 con 0x7f82fc13f9a0 2024-03-29T11:51:02.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.400+0000 7f8300887640 1 -- 172.21.15.98:0/3146449179 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1026293125 0 0) 0x7f82fc0b0030 con 0x7f82fc071f40 2024-03-29T11:51:02.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.400+0000 7f8300887640 10 monclient(hunting): _negotiate 2024-03-29T11:51:02.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.400+0000 7f8300887640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:02.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.400+0000 7f8300887640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:02.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.400+0000 7f8300887640 10 monclient(hunting): my global_id is 4348 2024-03-29T11:51:02.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.400+0000 7f8300887640 1 -- 172.21.15.98:0/3146449179 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82e8003840 con 0x7f82fc071f40 2024-03-29T11:51:02.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.400+0000 7f8300887640 1 -- 172.21.15.98:0/3146449179 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3368249313 0 0) 0x7f82e8003520 con 0x7f82fc13f9a0 2024-03-29T11:51:02.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.400+0000 7f8300887640 1 -- 172.21.15.98:0/3146449179 >> v1:172.21.15.121:6789/0 conn(0x7f82fc071f40 legacy=0x7f82fc13f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:02.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.401+0000 7f8300887640 1 -- 172.21.15.98:0/3146449179 >> v1:172.21.15.98:6789/0 conn(0x7f82fc0adcb0 legacy=0x7f82fc1457a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:51:02.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.401+0000 7f8300887640 10 monclient: _finish_hunting 0 2024-03-29T11:51:02.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.401+0000 7f8300887640 1 monclient: found mon.noname-b 2024-03-29T11:51:02.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.401+0000 7f8300887640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:51:02.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.401+0000 7f8300887640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:51:02.411 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.401+0000 7f8300887640 1 -- 172.21.15.98:0/3146449179 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82fc0aedd0 con 0x7f82fc13f9a0 2024-03-29T11:51:02.411 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.401+0000 7f8300887640 10 monclient: _finish_auth 0 2024-03-29T11:51:02.411 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.401+0000 7f8300887640 10 monclient: _check_auth_tickets 2024-03-29T11:51:02.411 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.401+0000 7f8300887640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:51:02.411 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.401+0000 7f8300887640 1 -- 172.21.15.98:0/3146449179 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f82f40019f0 con 0x7f82fc13f9a0 2024-03-29T11:51:02.411 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.401+0000 7f8300887640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:02.411 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.401+0000 7f8300887640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:51:02.411 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.401+0000 7f8300887640 10 monclient: dump: 2024-03-29T11:51:02.411 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:51:02.411 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:02.411 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:02.411 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:02.411 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:02.411 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:02.411 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:51:02.411 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:51:02.412 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:51:02.412 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:02.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.402+0000 7f8300887640 1 -- 172.21.15.98:0/3146449179 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82f4001bd0 con 0x7f82fc13f9a0 2024-03-29T11:51:02.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.402+0000 7f8300887640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:51:02.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.402+0000 7f8300887640 1 -- 172.21.15.98:0/3146449179 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f82f4001fe0 con 0x7f82fc13f9a0 2024-03-29T11:51:02.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.402+0000 7f8300887640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:02.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.402+0000 7f8300887640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:51:02.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.402+0000 7f8300887640 10 monclient: dump: 2024-03-29T11:51:02.412 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:51:02.412 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:02.412 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:02.412 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:02.412 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:02.412 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:02.412 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:51:02.412 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:51:02.413 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:51:02.413 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:02.413 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.402+0000 7f830288b640 5 monclient: authenticate success, global_id 4325 2024-03-29T11:51:02.413 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.402+0000 7f830288b640 10 monclient: get_monmap_and_config success 2024-03-29T11:51:02.413 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.402+0000 7f830288b640 4 set_mon_vals no callback set 2024-03-29T11:51:02.413 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.403+0000 7f830288b640 10 monclient: shutdown 2024-03-29T11:51:02.413 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.403+0000 7f830288b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:51:02.413 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.403+0000 7f830288b640 1 -- 172.21.15.98:0/3146449179 >> v1:172.21.15.98:6790/0 conn(0x7f82fc13f9a0 legacy=0x7f82fc145eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:02.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.403+0000 7f830288b640 1 -- 172.21.15.98:0/3146449179 shutdown_connections 2024-03-29T11:51:02.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.403+0000 7f830288b640 1 -- 172.21.15.98:0/3146449179 >> 172.21.15.98:0/3146449179 conn(0x7f82fc0673e0 msgr2=0x7f82fc0b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:51:02.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.404+0000 7f830288b640 1 -- 172.21.15.98:0/3146449179 shutdown_connections 2024-03-29T11:51:02.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.405+0000 7f830288b640 1 -- 172.21.15.98:0/3146449179 wait complete. 2024-03-29T11:51:02.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.406+0000 7f830288b640 10 monclient: build_initial_monmap 2024-03-29T11:51:02.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.406+0000 7f830288b640 10 monclient: monmap: 2024-03-29T11:51:02.415 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:51:02.415 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:51:02.415 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:51:02.415 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:51:02.415 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:51:02.415 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:02.415 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:51:02.415 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:51:02.415 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:51:02.415 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:02.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.406+0000 7f830288b640 1 librados: starting msgr at 2024-03-29T11:51:02.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.406+0000 7f830288b640 1 librados: starting objecter 2024-03-29T11:51:02.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.407+0000 7f830288b640 1 Processor -- start 2024-03-29T11:51:02.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.407+0000 7f830288b640 1 -- start start 2024-03-29T11:51:02.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.407+0000 7f830288b640 1 librados: setting wanted keys 2024-03-29T11:51:02.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.407+0000 7f830288b640 1 librados: calling monclient init 2024-03-29T11:51:02.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.407+0000 7f830288b640 10 monclient: init 2024-03-29T11:51:02.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.408+0000 7f830288b640 10 monclient: _reopen_session rank -1 2024-03-29T11:51:02.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.408+0000 7f830288b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:51:02.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.408+0000 7f830288b640 10 monclient(hunting): picked mon.noname-c con 0x7f82fc071f40 addr v1:172.21.15.98:6790/0 2024-03-29T11:51:02.417 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.408+0000 7f830288b640 10 monclient(hunting): picked mon.noname-b con 0x7f82fc0adcb0 addr v1:172.21.15.121:6789/0 2024-03-29T11:51:02.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.408+0000 7f830288b640 10 monclient(hunting): picked mon.noname-a con 0x7f82fc13f9a0 addr v1:172.21.15.98:6789/0 2024-03-29T11:51:02.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.408+0000 7f830288b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82fc12fb20 con 0x7f82fc13f9a0 2024-03-29T11:51:02.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.408+0000 7f830288b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82fc140280 con 0x7f82fc0adcb0 2024-03-29T11:51:02.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.408+0000 7f830288b640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82fc140560 con 0x7f82fc071f40 2024-03-29T11:51:02.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.408+0000 7f830288b640 10 monclient(hunting): _renew_subs 2024-03-29T11:51:02.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.408+0000 7f830288b640 10 monclient(hunting): authenticate will time out at 969.493261s 2024-03-29T11:51:02.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.408+0000 7f8301088640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f82fc0adcb0 0x7f82fc132840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:43400/0 (socket says 172.21.15.98:43400) 2024-03-29T11:51:02.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.408+0000 7f8301088640 1 -- 172.21.15.98:0/3438827653 learned_addr learned my addr 172.21.15.98:0/3438827653 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:51:02.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.408+0000 7f82f27fc640 10 client.?.objecter ms_handle_connect 0x7f82fc0adcb0 2024-03-29T11:51:02.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.408+0000 7f82f27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:51:02.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.409+0000 7f82f27fc640 1 -- 172.21.15.98:0/3438827653 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1549975312 0 0) 0x7f82fc140280 con 0x7f82fc0adcb0 2024-03-29T11:51:02.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.409+0000 7f82f27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:51:02.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.409+0000 7f82f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:02.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.409+0000 7f82f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:02.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.409+0000 7f82f27fc640 10 monclient(hunting): my global_id is 4351 2024-03-29T11:51:02.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.409+0000 7f82f27fc640 1 -- 172.21.15.98:0/3438827653 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82e0004eb0 con 0x7f82fc0adcb0 2024-03-29T11:51:02.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.410+0000 7f82f27fc640 1 -- 172.21.15.98:0/3438827653 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2047990835 0 0) 0x7f82e0004eb0 con 0x7f82fc0adcb0 2024-03-29T11:51:02.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.410+0000 7f82f27fc640 1 -- 172.21.15.98:0/3438827653 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82fc140280 con 0x7f82fc0adcb0 2024-03-29T11:51:02.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.410+0000 7f82f27fc640 1 -- 172.21.15.98:0/3438827653 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f82ec001960 con 0x7f82fc0adcb0 2024-03-29T11:51:02.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.410+0000 7f82f27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:02.420 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.410+0000 7f82f27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:51:02.420 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.410+0000 7f82f27fc640 10 monclient(hunting): dump: 2024-03-29T11:51:02.420 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:51:02.420 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:02.420 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:02.420 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:02.420 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:02.420 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:02.420 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:51:02.420 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:51:02.420 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:51:02.420 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:02.420 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.410+0000 7f82f27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:51:02.420 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.411+0000 7f82f27fc640 1 -- 172.21.15.98:0/3438827653 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3552349452 0 0) 0x7f82fc140280 con 0x7f82fc0adcb0 2024-03-29T11:51:02.421 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.411+0000 7f82f27fc640 1 -- 172.21.15.98:0/3438827653 >> v1:172.21.15.98:6790/0 conn(0x7f82fc071f40 legacy=0x7f82fc140c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:02.421 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.411+0000 7f82f27fc640 1 -- 172.21.15.98:0/3438827653 >> v1:172.21.15.98:6789/0 conn(0x7f82fc13f9a0 legacy=0x7f82fc1e2120 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:51:02.421 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.413+0000 7f82f27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:51:02.421 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.413+0000 7f82f27fc640 1 monclient: found mon.b 2024-03-29T11:51:02.421 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.413+0000 7f82f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:51:02.421 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.413+0000 7f82f27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:51:02.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.413+0000 7f82f27fc640 1 -- 172.21.15.98:0/3438827653 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82fc133350 con 0x7f82fc0adcb0 2024-03-29T11:51:02.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.413+0000 7f82f27fc640 10 monclient: _finish_auth 0 2024-03-29T11:51:02.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.413+0000 7f82f27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:51:02.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.413+0000 7f82f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:51:02.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.413+0000 7f82f27fc640 1 -- 172.21.15.98:0/3438827653 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82ec001ac0 con 0x7f82fc0adcb0 2024-03-29T11:51:02.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.413+0000 7f82f27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:51:02.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.413+0000 7f82f27fc640 1 -- 172.21.15.98:0/3438827653 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f82ec003a40 con 0x7f82fc0adcb0 2024-03-29T11:51:02.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.413+0000 7f82f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:02.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.413+0000 7f82f27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:51:02.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.413+0000 7f82f27fc640 10 monclient: dump: 2024-03-29T11:51:02.422 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:51:02.422 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:02.422 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:02.422 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:02.423 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:02.423 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:02.423 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:51:02.423 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:51:02.423 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:51:02.423 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:02.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.415+0000 7f830288b640 5 monclient: authenticate success, global_id 4351 2024-03-29T11:51:02.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.415+0000 7f82f3fff640 4 set_mon_vals no callback set 2024-03-29T11:51:02.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.415+0000 7f830288b640 10 monclient: _renew_subs 2024-03-29T11:51:02.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.416+0000 7f830288b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:51:02.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.416+0000 7f830288b640 1 -- 172.21.15.98:0/3438827653 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82fc1318e0 con 0x7f82fc0adcb0 2024-03-29T11:51:02.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.416+0000 7f830288b640 10 client.4351.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:51:02.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.416+0000 7f830288b640 10 monclient: _renew_subs 2024-03-29T11:51:02.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.416+0000 7f830288b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:51:02.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.416+0000 7f830288b640 1 -- 172.21.15.98:0/3438827653 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f82fc144800 con 0x7f82fc0adcb0 2024-03-29T11:51:02.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.416+0000 7f830288b640 1 librados: init done 2024-03-29T11:51:02.426 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.419+0000 7f82f27fc640 1 -- 172.21.15.98:0/3438827653 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f82ec003ba0 con 0x7f82fc0adcb0 2024-03-29T11:51:02.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.419+0000 7f830288b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:02.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.419+0000 7f830288b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:02.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.419+0000 7f830288b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:51:02.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.419+0000 7f830288b640 1 -- 172.21.15.98:0/3438827653 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82fc132560 con 0x7f82fc0adcb0 2024-03-29T11:51:02.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.419+0000 7f8301889640 10 monclient: get_auth_request con 0x7f82e006da70 auth_method 0 2024-03-29T11:51:02.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.420+0000 7f82f27fc640 1 -- 172.21.15.98:0/3438827653 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f82ec084a90 con 0x7f82fc0adcb0 2024-03-29T11:51:02.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.420+0000 7f82f27fc640 10 client.4351.objecter ms_dispatch 0x7f82fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:51:02.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.420+0000 7f82f27fc640 3 client.4351.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:51:02.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.420+0000 7f82f27fc640 3 client.4351.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:51:02.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.420+0000 7f82f27fc640 20 client.4351.objecter dump_active .. 0 homeless 2024-03-29T11:51:02.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.420+0000 7f82f27fc640 10 client.4351.objecter ms_handle_connect 0x7f82e006da70 2024-03-29T11:51:02.432 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.426+0000 7f82f27fc640 1 -- 172.21.15.98:0/3438827653 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f82fc132560 con 0x7f82fc0adcb0 2024-03-29T11:51:02.432 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.426+0000 7f82f27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:02.433 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.426+0000 7f82f27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:51:02.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.742+0000 7f830288b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:02.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.742+0000 7f830288b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:02.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.742+0000 7f830288b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:51:02.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.742+0000 7f830288b640 1 -- 172.21.15.98:0/3438827653 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82fc132560 con 0x7f82fc0adcb0 2024-03-29T11:51:02.750 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.743+0000 7f82f27fc640 1 -- 172.21.15.98:0/3438827653 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f82fc132560 con 0x7f82fc0adcb0 2024-03-29T11:51:02.750 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.743+0000 7f82f27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:02.750 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.743+0000 7f82f27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:51:02.750 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:51:02.751 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:51:02.753 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.746+0000 7f830288b640 10 librados: watch_flush enter 2024-03-29T11:51:02.753 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.746+0000 7f830288b640 10 librados: watch_flush exit 2024-03-29T11:51:02.753 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.746+0000 7f830288b640 20 client.4351.objecter shutdown clearing up homeless session... 2024-03-29T11:51:02.753 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.746+0000 7f830288b640 10 client.4351.objecter successfully canceled tick 2024-03-29T11:51:02.753 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.746+0000 7f830288b640 1 -- 172.21.15.98:0/3438827653 >> v1:172.21.15.98:6816/4065186974 conn(0x7f82e006da70 legacy=0x7f82e006ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:02.753 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.746+0000 7f830288b640 10 monclient: shutdown 2024-03-29T11:51:02.753 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.746+0000 7f830288b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:51:02.754 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.746+0000 7f830288b640 1 -- 172.21.15.98:0/3438827653 >> v1:172.21.15.121:6789/0 conn(0x7f82fc0adcb0 legacy=0x7f82fc132840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:02.756 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.748+0000 7f830288b640 1 -- 172.21.15.98:0/3438827653 shutdown_connections 2024-03-29T11:51:02.756 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.748+0000 7f830288b640 1 -- 172.21.15.98:0/3438827653 >> 172.21.15.98:0/3438827653 conn(0x7f82fc0673e0 msgr2=0x7f82fc069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:51:02.756 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.748+0000 7f830288b640 1 -- 172.21.15.98:0/3438827653 shutdown_connections 2024-03-29T11:51:02.756 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.748+0000 7f830288b640 1 -- 172.21.15.98:0/3438827653 wait complete. 2024-03-29T11:51:02.756 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:02.748+0000 7f830288b640 1 librados: shutdown 2024-03-29T11:51:02.772 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:51:09.579 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:51:09.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.722+0000 7f89eadeb640 10 monclient: get_monmap_and_config 2024-03-29T11:51:09.731 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.722+0000 7f89eadeb640 10 monclient: build_initial_monmap 2024-03-29T11:51:09.731 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.722+0000 7f89eadeb640 10 monclient: monmap: 2024-03-29T11:51:09.731 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:51:09.731 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:09.731 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:51:09.723204+0000 2024-03-29T11:51:09.731 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:51:09.723204+0000 2024-03-29T11:51:09.731 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:51:09.731 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:09.731 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:51:09.731 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:51:09.731 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:51:09.731 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:09.736 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.725+0000 7f89eadeb640 1 Processor -- start 2024-03-29T11:51:09.736 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.725+0000 7f89eadeb640 1 -- start start 2024-03-29T11:51:09.737 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.725+0000 7f89eadeb640 10 monclient: init 2024-03-29T11:51:09.737 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.726+0000 7f89eadeb640 10 monclient: _reopen_session rank -1 2024-03-29T11:51:09.737 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.726+0000 7f89eadeb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:51:09.737 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.726+0000 7f89eadeb640 10 monclient(hunting): picked mon.noname-b con 0x7f89dc034850 addr v1:172.21.15.98:6790/0 2024-03-29T11:51:09.737 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.726+0000 7f89eadeb640 10 monclient(hunting): picked mon.noname-a con 0x7f89dc02a640 addr v1:172.21.15.98:6789/0 2024-03-29T11:51:09.737 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.726+0000 7f89eadeb640 10 monclient(hunting): picked mon.noname-c con 0x7f89dc0327d0 addr v1:172.21.15.121:6789/0 2024-03-29T11:51:09.737 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.726+0000 7f89eadeb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89dc02c820 con 0x7f89dc02a640 2024-03-29T11:51:09.737 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.726+0000 7f89eadeb640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89dc02a120 con 0x7f89dc0327d0 2024-03-29T11:51:09.737 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.726+0000 7f89eadeb640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89dc0c75b0 con 0x7f89dc034850 2024-03-29T11:51:09.737 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.726+0000 7f89eadeb640 10 monclient(hunting): _renew_subs 2024-03-29T11:51:09.737 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.726+0000 7f89eadeb640 10 monclient(hunting): authenticate will time out at 976.811852s 2024-03-29T11:51:09.737 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.730+0000 7f89ea5ea640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f89dc0327d0 0x7f89dc0cf240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:57607/0 (socket says 172.21.15.98:57607) 2024-03-29T11:51:09.737 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.730+0000 7f89ea5ea640 1 -- 172.21.15.98:0/3383874593 learned_addr learned my addr 172.21.15.98:0/3383874593 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:51:09.739 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.730+0000 7f89e8de7640 1 -- 172.21.15.98:0/3383874593 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2286010234 0 0) 0x7f89dc02c820 con 0x7f89dc02a640 2024-03-29T11:51:09.739 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.730+0000 7f89e8de7640 10 monclient(hunting): _negotiate 2024-03-29T11:51:09.739 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.730+0000 7f89e8de7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:09.739 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.730+0000 7f89e8de7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:09.739 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.730+0000 7f89e8de7640 10 monclient(hunting): my global_id is 4320 2024-03-29T11:51:09.739 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.730+0000 7f89e8de7640 1 -- 172.21.15.98:0/3383874593 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89d0003520 con 0x7f89dc02a640 2024-03-29T11:51:09.739 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.730+0000 7f89e8de7640 1 -- 172.21.15.98:0/3383874593 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 10297182 0 0) 0x7f89dc02a120 con 0x7f89dc0327d0 2024-03-29T11:51:09.739 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.730+0000 7f89e8de7640 10 monclient(hunting): _negotiate 2024-03-29T11:51:09.739 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.730+0000 7f89e8de7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:09.739 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.730+0000 7f89e8de7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:09.739 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.730+0000 7f89e8de7640 10 monclient(hunting): my global_id is 4354 2024-03-29T11:51:09.739 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.730+0000 7f89e8de7640 1 -- 172.21.15.98:0/3383874593 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89dc02c820 con 0x7f89dc0327d0 2024-03-29T11:51:09.740 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.731+0000 7f89e8de7640 1 -- 172.21.15.98:0/3383874593 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1869431924 0 0) 0x7f89d0003520 con 0x7f89dc02a640 2024-03-29T11:51:09.740 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.731+0000 7f89e8de7640 1 -- 172.21.15.98:0/3383874593 >> v1:172.21.15.98:6790/0 conn(0x7f89dc034850 legacy=0x7f89dc0c6ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:51:09.744 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.735+0000 7f89e8de7640 1 -- 172.21.15.98:0/3383874593 >> v1:172.21.15.121:6789/0 conn(0x7f89dc0327d0 legacy=0x7f89dc0cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:09.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.736+0000 7f89e8de7640 10 monclient: _finish_hunting 0 2024-03-29T11:51:09.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.736+0000 7f89e8de7640 1 monclient: found mon.noname-a 2024-03-29T11:51:09.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.736+0000 7f89e8de7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:51:09.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.736+0000 7f89e8de7640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T11:51:09.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.736+0000 7f89e8de7640 1 -- 172.21.15.98:0/3383874593 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89dc033110 con 0x7f89dc02a640 2024-03-29T11:51:09.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.736+0000 7f89e8de7640 10 monclient: _finish_auth 0 2024-03-29T11:51:09.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.736+0000 7f89e8de7640 10 monclient: _check_auth_tickets 2024-03-29T11:51:09.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.736+0000 7f89e8de7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:51:09.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.736+0000 7f89eadeb640 5 monclient: authenticate success, global_id 4320 2024-03-29T11:51:09.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.736+0000 7f89eadeb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:51:09.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.736+0000 7f89e8de7640 1 -- 172.21.15.98:0/3383874593 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f89e0001690 con 0x7f89dc02a640 2024-03-29T11:51:09.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.736+0000 7f89e8de7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:09.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.736+0000 7f89e8de7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:51:09.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.736+0000 7f89e8de7640 10 monclient: dump: 2024-03-29T11:51:09.746 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:51:09.746 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:09.746 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:09.746 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:09.746 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:09.746 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:09.746 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:51:09.746 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:51:09.746 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:51:09.746 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:09.746 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.736+0000 7f89eadeb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:51:09.746 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.736+0000 7f89e8de7640 1 -- 172.21.15.98:0/3383874593 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89e0001860 con 0x7f89dc02a640 2024-03-29T11:51:09.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.736+0000 7f89e8de7640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:51:09.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.736+0000 7f89eadeb640 10 monclient: get_monmap_and_config success 2024-03-29T11:51:09.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.736+0000 7f89eadeb640 4 set_mon_vals no callback set 2024-03-29T11:51:09.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.737+0000 7f89eadeb640 10 monclient: shutdown 2024-03-29T11:51:09.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.737+0000 7f89eadeb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:51:09.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.737+0000 7f89eadeb640 1 -- 172.21.15.98:0/3383874593 >> v1:172.21.15.98:6789/0 conn(0x7f89dc02a640 legacy=0x7f89dc0320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:09.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.737+0000 7f89eadeb640 1 -- 172.21.15.98:0/3383874593 shutdown_connections 2024-03-29T11:51:09.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.737+0000 7f89eadeb640 1 -- 172.21.15.98:0/3383874593 >> 172.21.15.98:0/3383874593 conn(0x7f89dc0058f0 msgr2=0x7f89dc005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:51:09.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.737+0000 7f89eadeb640 1 -- 172.21.15.98:0/3383874593 shutdown_connections 2024-03-29T11:51:09.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.737+0000 7f89eadeb640 1 -- 172.21.15.98:0/3383874593 wait complete. 2024-03-29T11:51:09.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.738+0000 7f89eadeb640 10 monclient: build_initial_monmap 2024-03-29T11:51:09.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.738+0000 7f89eadeb640 10 monclient: monmap: 2024-03-29T11:51:09.748 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:51:09.748 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:51:09.748 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:51:09.748 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:51:09.748 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:51:09.748 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:09.748 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:51:09.748 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:51:09.748 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:51:09.749 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:09.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.739+0000 7f89eadeb640 1 librados: starting msgr at 2024-03-29T11:51:09.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.739+0000 7f89eadeb640 1 librados: starting objecter 2024-03-29T11:51:09.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.739+0000 7f89eadeb640 1 Processor -- start 2024-03-29T11:51:09.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.739+0000 7f89eadeb640 1 -- start start 2024-03-29T11:51:09.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.739+0000 7f89eadeb640 1 librados: setting wanted keys 2024-03-29T11:51:09.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.739+0000 7f89eadeb640 1 librados: calling monclient init 2024-03-29T11:51:09.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.739+0000 7f89eadeb640 10 monclient: init 2024-03-29T11:51:09.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.740+0000 7f89eadeb640 10 monclient: _reopen_session rank -1 2024-03-29T11:51:09.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.740+0000 7f89eadeb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:51:09.752 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.742+0000 7f89eadeb640 10 monclient(hunting): picked mon.noname-c con 0x7f89dc0327d0 addr v1:172.21.15.98:6790/0 2024-03-29T11:51:09.752 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.742+0000 7f89eadeb640 10 monclient(hunting): picked mon.noname-a con 0x7f89dc034850 addr v1:172.21.15.98:6789/0 2024-03-29T11:51:09.752 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.742+0000 7f89eadeb640 10 monclient(hunting): picked mon.noname-b con 0x7f89dc0c3b40 addr v1:172.21.15.121:6789/0 2024-03-29T11:51:09.752 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.742+0000 7f89eadeb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89dc0ba810 con 0x7f89dc034850 2024-03-29T11:51:09.753 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.742+0000 7f89eadeb640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89dc0baa80 con 0x7f89dc0c3b40 2024-03-29T11:51:09.753 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.742+0000 7f89eadeb640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89dc0baea0 con 0x7f89dc0327d0 2024-03-29T11:51:09.753 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.742+0000 7f89eadeb640 10 monclient(hunting): _renew_subs 2024-03-29T11:51:09.753 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.742+0000 7f89eadeb640 10 monclient(hunting): authenticate will time out at 976.827721s 2024-03-29T11:51:09.753 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.744+0000 7f89e9de9640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f89dc0327d0 0x7f89dc0bb430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:32873/0 (socket says 172.21.15.98:32873) 2024-03-29T11:51:09.753 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.744+0000 7f89e9de9640 1 -- 172.21.15.98:0/1813627178 learned_addr learned my addr 172.21.15.98:0/1813627178 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:51:09.753 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.744+0000 7f89da7fc640 10 client.?.objecter ms_handle_connect 0x7f89dc0327d0 2024-03-29T11:51:09.753 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.744+0000 7f89da7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:51:09.753 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.745+0000 7f89da7fc640 1 -- 172.21.15.98:0/1813627178 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 237730275 0 0) 0x7f89dc0baea0 con 0x7f89dc0327d0 2024-03-29T11:51:09.753 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.745+0000 7f89da7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:51:09.753 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.745+0000 7f89da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:09.753 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.745+0000 7f89da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:09.753 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.745+0000 7f89da7fc640 10 monclient(hunting): my global_id is 4328 2024-03-29T11:51:09.753 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.745+0000 7f89da7fc640 1 -- 172.21.15.98:0/1813627178 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89c8004eb0 con 0x7f89dc0327d0 2024-03-29T11:51:09.753 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.745+0000 7f89da7fc640 1 -- 172.21.15.98:0/1813627178 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3956919135 0 0) 0x7f89c8004eb0 con 0x7f89dc0327d0 2024-03-29T11:51:09.754 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.746+0000 7f89da7fc640 1 -- 172.21.15.98:0/1813627178 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89dc0baea0 con 0x7f89dc0327d0 2024-03-29T11:51:09.754 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.746+0000 7f89da7fc640 1 -- 172.21.15.98:0/1813627178 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f89e404fdd0 con 0x7f89dc0327d0 2024-03-29T11:51:09.754 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.746+0000 7f89da7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:09.754 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.746+0000 7f89da7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:51:09.754 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.746+0000 7f89da7fc640 10 monclient(hunting): dump: 2024-03-29T11:51:09.754 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:51:09.754 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:09.754 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:09.754 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:09.754 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:09.755 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:09.755 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:51:09.755 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:51:09.755 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:51:09.755 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:09.755 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.746+0000 7f89da7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:51:09.755 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.746+0000 7f89da7fc640 1 -- 172.21.15.98:0/1813627178 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1419770822 0 0) 0x7f89dc0baea0 con 0x7f89dc0327d0 2024-03-29T11:51:09.755 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.746+0000 7f89da7fc640 1 -- 172.21.15.98:0/1813627178 >> v1:172.21.15.121:6789/0 conn(0x7f89dc0c3b40 legacy=0x7f89dc173cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:09.755 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.746+0000 7f89da7fc640 1 -- 172.21.15.98:0/1813627178 >> v1:172.21.15.98:6789/0 conn(0x7f89dc034850 legacy=0x7f89dc0bbb40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:51:09.756 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.747+0000 7f89da7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:51:09.756 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.747+0000 7f89da7fc640 1 monclient: found mon.c 2024-03-29T11:51:09.756 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.747+0000 7f89da7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:51:09.756 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.747+0000 7f89da7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:51:09.756 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.747+0000 7f89da7fc640 1 -- 172.21.15.98:0/1813627178 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89dc0c2730 con 0x7f89dc0327d0 2024-03-29T11:51:09.756 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.747+0000 7f89da7fc640 10 monclient: _finish_auth 0 2024-03-29T11:51:09.756 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.747+0000 7f89da7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:51:09.757 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.747+0000 7f89da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:51:09.757 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.747+0000 7f89eadeb640 5 monclient: authenticate success, global_id 4328 2024-03-29T11:51:09.757 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.747+0000 7f89eadeb640 10 monclient: _renew_subs 2024-03-29T11:51:09.757 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.747+0000 7f89eadeb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:51:09.757 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.747+0000 7f89eadeb640 1 -- 172.21.15.98:0/1813627178 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89dc0bfe50 con 0x7f89dc0327d0 2024-03-29T11:51:09.757 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.747+0000 7f89eadeb640 10 client.4328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:51:09.757 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.747+0000 7f89eadeb640 10 monclient: _renew_subs 2024-03-29T11:51:09.757 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.747+0000 7f89eadeb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:51:09.757 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.747+0000 7f89eadeb640 1 -- 172.21.15.98:0/1813627178 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f89dc1753d0 con 0x7f89dc0327d0 2024-03-29T11:51:09.757 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.747+0000 7f89eadeb640 1 librados: init done 2024-03-29T11:51:09.757 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.749+0000 7f89cffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:09.757 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.749+0000 7f89cffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:09.757 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.749+0000 7f89cffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:51:09.758 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.749+0000 7f89cffff640 1 -- 172.21.15.98:0/1813627178 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89dc004cd0 con 0x7f89dc0327d0 2024-03-29T11:51:09.758 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.751+0000 7f89da7fc640 1 -- 172.21.15.98:0/1813627178 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89e404f400 con 0x7f89dc0327d0 2024-03-29T11:51:09.758 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.751+0000 7f89da7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:51:09.758 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.751+0000 7f89da7fc640 1 -- 172.21.15.98:0/1813627178 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f89e4075f10 con 0x7f89dc0327d0 2024-03-29T11:51:09.758 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.751+0000 7f89da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:09.758 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.751+0000 7f89da7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:51:09.758 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.751+0000 7f89da7fc640 10 monclient: dump: 2024-03-29T11:51:09.758 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:51:09.758 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:09.759 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:09.759 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:09.759 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:09.759 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:09.759 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:51:09.759 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:51:09.759 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:51:09.759 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:09.759 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.751+0000 7f89dbfff640 4 set_mon_vals no callback set 2024-03-29T11:51:09.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.751+0000 7f89da7fc640 1 -- 172.21.15.98:0/1813627178 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f89e4067bf0 con 0x7f89dc0327d0 2024-03-29T11:51:09.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.753+0000 7f89da7fc640 1 -- 172.21.15.98:0/1813627178 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f89e40ff600 con 0x7f89dc0327d0 2024-03-29T11:51:09.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.753+0000 7f89da7fc640 10 client.4328.objecter ms_dispatch 0x7f89e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:51:09.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.753+0000 7f89da7fc640 3 client.4328.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:51:09.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.753+0000 7f89da7fc640 3 client.4328.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:51:09.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.753+0000 7f89da7fc640 20 client.4328.objecter dump_active .. 0 homeless 2024-03-29T11:51:09.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.754+0000 7f89e95e8640 10 monclient: get_auth_request con 0x7f89c806db50 auth_method 0 2024-03-29T11:51:09.761 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.754+0000 7f89da7fc640 10 client.4328.objecter ms_handle_connect 0x7f89c806db50 2024-03-29T11:51:09.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.759+0000 7f89da7fc640 1 -- 172.21.15.98:0/1813627178 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f89e40cb030 con 0x7f89dc0327d0 2024-03-29T11:51:09.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.759+0000 7f89da7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:09.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:09.759+0000 7f89da7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:51:10.090 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:10.083+0000 7f89cffff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:10.090 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:10.083+0000 7f89cffff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:10.090 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:10.083+0000 7f89cffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:51:10.091 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:10.083+0000 7f89cffff640 1 -- 172.21.15.98:0/1813627178 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f89dc0008d0 con 0x7f89dc0327d0 2024-03-29T11:51:10.091 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:10.084+0000 7f89da7fc640 1 -- 172.21.15.98:0/1813627178 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f89e40ccf60 con 0x7f89dc0327d0 2024-03-29T11:51:10.091 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:10.084+0000 7f89da7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:10.091 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:10.084+0000 7f89da7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:51:10.092 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:51:10.092 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:51:10.096 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:10.088+0000 7f89cffff640 10 librados: watch_flush enter 2024-03-29T11:51:10.096 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:10.090+0000 7f89cffff640 10 librados: watch_flush exit 2024-03-29T11:51:10.097 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:10.090+0000 7f89cffff640 20 client.4328.objecter shutdown clearing up homeless session... 2024-03-29T11:51:10.097 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:10.090+0000 7f89cffff640 10 client.4328.objecter successfully canceled tick 2024-03-29T11:51:10.098 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:10.091+0000 7f89cffff640 1 -- 172.21.15.98:0/1813627178 >> v1:172.21.15.98:6816/4065186974 conn(0x7f89c806db50 legacy=0x7f89c8070010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:10.099 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:10.091+0000 7f89cffff640 10 monclient: shutdown 2024-03-29T11:51:10.099 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:10.091+0000 7f89cffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:51:10.099 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:10.091+0000 7f89cffff640 1 -- 172.21.15.98:0/1813627178 >> v1:172.21.15.98:6790/0 conn(0x7f89dc0327d0 legacy=0x7f89dc0bb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:10.099 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:10.092+0000 7f89cffff640 1 -- 172.21.15.98:0/1813627178 shutdown_connections 2024-03-29T11:51:10.099 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:10.092+0000 7f89cffff640 1 -- 172.21.15.98:0/1813627178 >> 172.21.15.98:0/1813627178 conn(0x7f89dc0058f0 msgr2=0x7f89dc0043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:51:10.099 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:10.092+0000 7f89cffff640 1 -- 172.21.15.98:0/1813627178 shutdown_connections 2024-03-29T11:51:10.099 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:10.092+0000 7f89cffff640 1 -- 172.21.15.98:0/1813627178 wait complete. 2024-03-29T11:51:10.099 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:10.092+0000 7f89cffff640 1 librados: shutdown 2024-03-29T11:51:10.119 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:51:16.928 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:51:17.078 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.070+0000 7f62831f4640 10 monclient: get_monmap_and_config 2024-03-29T11:51:17.078 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.070+0000 7f62831f4640 10 monclient: build_initial_monmap 2024-03-29T11:51:17.078 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.070+0000 7f62831f4640 10 monclient: monmap: 2024-03-29T11:51:17.078 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:51:17.078 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:17.078 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:51:17.071095+0000 2024-03-29T11:51:17.078 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:51:17.071095+0000 2024-03-29T11:51:17.079 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:51:17.079 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:17.079 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:51:17.079 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:51:17.079 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:51:17.079 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:17.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.071+0000 7f62831f4640 1 Processor -- start 2024-03-29T11:51:17.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.071+0000 7f62831f4640 1 -- start start 2024-03-29T11:51:17.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.071+0000 7f62831f4640 10 monclient: init 2024-03-29T11:51:17.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.072+0000 7f62831f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:51:17.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.072+0000 7f62831f4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:51:17.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.072+0000 7f62831f4640 10 monclient(hunting): picked mon.noname-c con 0x7f627c0a9ac0 addr v1:172.21.15.121:6789/0 2024-03-29T11:51:17.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.072+0000 7f62831f4640 10 monclient(hunting): picked mon.noname-a con 0x7f627c13ee90 addr v1:172.21.15.98:6789/0 2024-03-29T11:51:17.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.072+0000 7f62831f4640 10 monclient(hunting): picked mon.noname-b con 0x7f627c0af900 addr v1:172.21.15.98:6790/0 2024-03-29T11:51:17.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.072+0000 7f62831f4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f627c0afd10 con 0x7f627c13ee90 2024-03-29T11:51:17.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.072+0000 7f62831f4640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f627c0af6f0 con 0x7f627c0a9ac0 2024-03-29T11:51:17.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.072+0000 7f62831f4640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f627c0aac80 con 0x7f627c0af900 2024-03-29T11:51:17.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.072+0000 7f62831f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:51:17.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.072+0000 7f62831f4640 10 monclient(hunting): authenticate will time out at 984.157853s 2024-03-29T11:51:17.082 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.075+0000 7f6281791640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f627c0af900 0x7f627c149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:40151/0 (socket says 172.21.15.98:40151) 2024-03-29T11:51:17.082 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.075+0000 7f6281791640 1 -- 172.21.15.98:0/2387279638 learned_addr learned my addr 172.21.15.98:0/2387279638 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:51:17.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.076+0000 7f627b7fe640 1 -- 172.21.15.98:0/2387279638 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1101695711 0 0) 0x7f627c0aac80 con 0x7f627c0af900 2024-03-29T11:51:17.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.076+0000 7f627b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:51:17.084 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.076+0000 7f627b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:17.084 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.076+0000 7f627b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:17.084 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.076+0000 7f627b7fe640 10 monclient(hunting): my global_id is 4331 2024-03-29T11:51:17.084 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.076+0000 7f627b7fe640 1 -- 172.21.15.98:0/2387279638 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6264003520 con 0x7f627c0af900 2024-03-29T11:51:17.084 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.076+0000 7f627b7fe640 1 -- 172.21.15.98:0/2387279638 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1602362409 0 0) 0x7f6264003520 con 0x7f627c0af900 2024-03-29T11:51:17.084 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.076+0000 7f627b7fe640 1 -- 172.21.15.98:0/2387279638 >> v1:172.21.15.121:6789/0 conn(0x7f627c0a9ac0 legacy=0x7f627c0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:51:17.084 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.076+0000 7f627b7fe640 1 -- 172.21.15.98:0/2387279638 >> v1:172.21.15.98:6789/0 conn(0x7f627c13ee90 legacy=0x7f627c13b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:51:17.090 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.080+0000 7f627b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:51:17.090 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.080+0000 7f627b7fe640 1 monclient: found mon.noname-b 2024-03-29T11:51:17.090 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.080+0000 7f627b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:51:17.090 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.081+0000 7f627b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:51:17.090 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.081+0000 7f627b7fe640 1 -- 172.21.15.98:0/2387279638 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f627c0aaf60 con 0x7f627c0af900 2024-03-29T11:51:17.090 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.081+0000 7f627b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:51:17.090 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.081+0000 7f627b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:51:17.090 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.081+0000 7f627b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:51:17.090 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.081+0000 7f627b7fe640 1 -- 172.21.15.98:0/2387279638 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f6270001690 con 0x7f627c0af900 2024-03-29T11:51:17.090 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.081+0000 7f62831f4640 5 monclient: authenticate success, global_id 4331 2024-03-29T11:51:17.090 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.081+0000 7f62831f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:51:17.090 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.081+0000 7f627b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:17.091 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.081+0000 7f6281791640 1 -- 172.21.15.98:0/2387279638 >> v1:172.21.15.98:6790/0 conn(0x7f627c0af900 legacy=0x7f627c149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-29T11:51:17.091 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.081+0000 7f6281791640 1 -- 172.21.15.98:0/2387279638 >> v1:172.21.15.98:6790/0 conn(0x7f627c0af900 legacy=0x7f627c149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:51:17.091 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.081+0000 7f6281791640 1 --1- 172.21.15.98:0/2387279638 >> v1:172.21.15.98:6790/0 conn(0x7f627c0af900 0x7f627c149b00 :-1 s=OPENED pgs=277571 cs=1 l=1).handle_message read tag failed 2024-03-29T11:51:17.091 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.081+0000 7f627b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:51:17.091 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.081+0000 7f6281791640 1 --1- 172.21.15.98:0/2387279638 >> v1:172.21.15.98:6790/0 conn(0x7f627c0af900 0x7f627c149b00 :-1 s=OPENED pgs=277571 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:51:17.091 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.081+0000 7f627b7fe640 10 monclient: dump: 2024-03-29T11:51:17.091 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:51:17.091 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:17.091 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:17.091 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:17.091 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:17.091 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:17.091 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:51:17.092 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:51:17.092 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:51:17.092 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:17.092 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.081+0000 7f62831f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:51:17.092 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.081+0000 7f627b7fe640 10 monclient: ms_handle_reset current mon v1:172.21.15.98:6790/0 2024-03-29T11:51:17.092 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.081+0000 7f627b7fe640 10 monclient: _reopen_session rank -1 2024-03-29T11:51:17.092 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.081+0000 7f627b7fe640 1 -- 172.21.15.98:0/2387279638 >> v1:172.21.15.98:6790/0 conn(0x7f627c0af900 legacy=0x7f627c149b00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:51:17.092 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.081+0000 7f627b7fe640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:51:17.092 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.082+0000 7f627b7fe640 10 monclient(hunting): picked mon.b con 0x7f627c0a9ac0 addr v1:172.21.15.121:6789/0 2024-03-29T11:51:17.092 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.082+0000 7f627b7fe640 10 monclient(hunting): picked mon.c con 0x7f6264004390 addr v1:172.21.15.98:6790/0 2024-03-29T11:51:17.092 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.082+0000 7f627b7fe640 10 monclient(hunting): picked mon.a con 0x7f627c13ee90 addr v1:172.21.15.98:6789/0 2024-03-29T11:51:17.092 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.082+0000 7f627b7fe640 1 -- 172.21.15.98:0/2387279638 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6264003520 con 0x7f627c13ee90 2024-03-29T11:51:17.092 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.082+0000 7f627b7fe640 1 -- 172.21.15.98:0/2387279638 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f627c0afd10 con 0x7f627c0a9ac0 2024-03-29T11:51:17.092 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.082+0000 7f627b7fe640 1 -- 172.21.15.98:0/2387279638 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f627c0af6f0 con 0x7f6264004390 2024-03-29T11:51:17.092 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.082+0000 7f627b7fe640 10 monclient(hunting): _renew_subs 2024-03-29T11:51:17.093 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.082+0000 7f627b7fe640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.98:6790/0 2024-03-29T11:51:17.098 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.089+0000 7f627b7fe640 1 -- 172.21.15.98:0/2387279638 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3439142522 0 0) 0x7f627c0af6f0 con 0x7f6264004390 2024-03-29T11:51:17.098 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.089+0000 7f627b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:51:17.098 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.089+0000 7f627b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:17.098 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.089+0000 7f627b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T11:51:17.098 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.089+0000 7f627b7fe640 1 -- 172.21.15.98:0/2387279638 --> v1:172.21.15.98:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f6264010d70 con 0x7f6264004390 2024-03-29T11:51:17.098 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.089+0000 7f627b7fe640 1 -- 172.21.15.98:0/2387279638 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1540854956 0 0) 0x7f6264003520 con 0x7f627c13ee90 2024-03-29T11:51:17.098 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.089+0000 7f627b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:51:17.098 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.089+0000 7f627b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:17.098 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.089+0000 7f627b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T11:51:17.098 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.089+0000 7f627b7fe640 1 -- 172.21.15.98:0/2387279638 --> v1:172.21.15.98:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f627c0af6f0 con 0x7f627c13ee90 2024-03-29T11:51:17.098 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.090+0000 7f627b7fe640 1 -- 172.21.15.98:0/2387279638 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 903209016 0 0) 0x7f627c0af6f0 con 0x7f627c13ee90 2024-03-29T11:51:17.099 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.090+0000 7f627b7fe640 1 -- 172.21.15.98:0/2387279638 >> v1:172.21.15.98:6790/0 conn(0x7f6264004390 legacy=0x7f6264009a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:17.099 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.090+0000 7f627b7fe640 1 -- 172.21.15.98:0/2387279638 >> v1:172.21.15.121:6789/0 conn(0x7f627c0a9ac0 legacy=0x7f6264003af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:17.099 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.090+0000 7f627b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:51:17.099 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.090+0000 7f627b7fe640 1 monclient: found mon.a 2024-03-29T11:51:17.099 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.090+0000 7f627b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:51:17.099 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.090+0000 7f627b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:51:17.099 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.090+0000 7f627b7fe640 1 -- 172.21.15.98:0/2387279638 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7f627c0aac80 con 0x7f627c13ee90 2024-03-29T11:51:17.099 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.090+0000 7f627b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:51:17.099 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.090+0000 7f627b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:51:17.099 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.090+0000 7f627b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:51:17.099 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.090+0000 7f627b7fe640 1 -- 172.21.15.98:0/2387279638 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f6270004870 con 0x7f627c13ee90 2024-03-29T11:51:17.099 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.090+0000 7f627b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:17.099 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.090+0000 7f627b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:51:17.099 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.090+0000 7f627b7fe640 10 monclient: dump: 2024-03-29T11:51:17.099 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:51:17.100 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:17.100 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:17.100 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:17.100 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:17.100 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:17.100 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:51:17.100 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:51:17.100 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:51:17.100 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:17.100 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.090+0000 7f627b7fe640 1 -- 172.21.15.98:0/2387279638 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6270004a50 con 0x7f627c13ee90 2024-03-29T11:51:17.100 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.090+0000 7f627b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:51:17.101 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.093+0000 7f62831f4640 10 monclient: get_monmap_and_config success 2024-03-29T11:51:17.101 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.093+0000 7f62831f4640 4 set_mon_vals no callback set 2024-03-29T11:51:17.101 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.094+0000 7f62831f4640 10 monclient: shutdown 2024-03-29T11:51:17.101 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.094+0000 7f62831f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:51:17.101 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.094+0000 7f62831f4640 1 -- 172.21.15.98:0/2387279638 >> v1:172.21.15.98:6789/0 conn(0x7f627c13ee90 legacy=0x7f626400d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:17.102 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.094+0000 7f62831f4640 1 -- 172.21.15.98:0/2387279638 shutdown_connections 2024-03-29T11:51:17.102 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.094+0000 7f62831f4640 1 -- 172.21.15.98:0/2387279638 >> 172.21.15.98:0/2387279638 conn(0x7f627c0b35e0 msgr2=0x7f627c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:51:17.102 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.094+0000 7f62831f4640 1 -- 172.21.15.98:0/2387279638 shutdown_connections 2024-03-29T11:51:17.102 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.094+0000 7f62831f4640 1 -- 172.21.15.98:0/2387279638 wait complete. 2024-03-29T11:51:17.108 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.100+0000 7f62831f4640 10 monclient: build_initial_monmap 2024-03-29T11:51:17.108 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.100+0000 7f62831f4640 10 monclient: monmap: 2024-03-29T11:51:17.108 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:51:17.108 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:51:17.108 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:51:17.108 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:51:17.108 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:51:17.108 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:17.108 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:51:17.108 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:51:17.108 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:51:17.108 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:17.108 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.101+0000 7f62831f4640 1 librados: starting msgr at 2024-03-29T11:51:17.109 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.101+0000 7f62831f4640 1 librados: starting objecter 2024-03-29T11:51:17.109 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.101+0000 7f62831f4640 1 Processor -- start 2024-03-29T11:51:17.113 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.101+0000 7f62831f4640 1 -- start start 2024-03-29T11:51:17.113 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.101+0000 7f62831f4640 1 librados: setting wanted keys 2024-03-29T11:51:17.113 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.101+0000 7f62831f4640 1 librados: calling monclient init 2024-03-29T11:51:17.113 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.101+0000 7f62831f4640 10 monclient: init 2024-03-29T11:51:17.113 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.102+0000 7f62831f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:51:17.113 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.102+0000 7f62831f4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:51:17.113 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.102+0000 7f62831f4640 10 monclient(hunting): picked mon.noname-a con 0x7f6264004390 addr v1:172.21.15.98:6789/0 2024-03-29T11:51:17.113 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.102+0000 7f62831f4640 10 monclient(hunting): picked mon.noname-c con 0x7f627c0a9ac0 addr v1:172.21.15.98:6790/0 2024-03-29T11:51:17.113 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.102+0000 7f62831f4640 10 monclient(hunting): picked mon.noname-b con 0x7f627c13ee90 addr v1:172.21.15.121:6789/0 2024-03-29T11:51:17.113 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.102+0000 7f62831f4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f627c1c90f0 con 0x7f6264004390 2024-03-29T11:51:17.113 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.102+0000 7f62831f4640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f627c1c74e0 con 0x7f627c13ee90 2024-03-29T11:51:17.113 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.102+0000 7f62831f4640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f627c1c68f0 con 0x7f627c0a9ac0 2024-03-29T11:51:17.114 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.102+0000 7f62831f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:51:17.114 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.102+0000 7f62831f4640 10 monclient(hunting): authenticate will time out at 984.187948s 2024-03-29T11:51:17.114 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.105+0000 7f6280f90640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f6264004390 0x7f627c071b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54207/0 (socket says 172.21.15.98:54207) 2024-03-29T11:51:17.114 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.105+0000 7f6280f90640 1 -- 172.21.15.98:0/3664793498 learned_addr learned my addr 172.21.15.98:0/3664793498 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:51:17.116 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.109+0000 7f62797fa640 10 client.?.objecter ms_handle_connect 0x7f6264004390 2024-03-29T11:51:17.116 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.109+0000 7f62797fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:51:17.116 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.109+0000 7f62797fa640 10 client.?.objecter ms_handle_connect 0x7f627c13ee90 2024-03-29T11:51:17.116 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.109+0000 7f62797fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:51:17.116 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.109+0000 7f62797fa640 1 -- 172.21.15.98:0/3664793498 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4152136730 0 0) 0x7f627c1c90f0 con 0x7f6264004390 2024-03-29T11:51:17.116 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.109+0000 7f62797fa640 10 monclient(hunting): _negotiate 2024-03-29T11:51:17.116 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.109+0000 7f62797fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:17.116 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.109+0000 7f62797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:17.116 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.109+0000 7f62797fa640 10 monclient(hunting): my global_id is 4323 2024-03-29T11:51:17.116 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.109+0000 7f62797fa640 1 -- 172.21.15.98:0/3664793498 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6268004eb0 con 0x7f6264004390 2024-03-29T11:51:17.116 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.109+0000 7f62797fa640 10 client.?.objecter ms_handle_connect 0x7f627c0a9ac0 2024-03-29T11:51:17.116 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.109+0000 7f62797fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:51:17.118 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.109+0000 7f62797fa640 1 -- 172.21.15.98:0/3664793498 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2718440739 0 0) 0x7f627c1c74e0 con 0x7f627c13ee90 2024-03-29T11:51:17.118 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.109+0000 7f62797fa640 10 monclient(hunting): _negotiate 2024-03-29T11:51:17.118 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.109+0000 7f62797fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:17.118 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.109+0000 7f62797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:17.118 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.109+0000 7f62797fa640 10 monclient(hunting): my global_id is 4357 2024-03-29T11:51:17.118 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.109+0000 7f62797fa640 1 -- 172.21.15.98:0/3664793498 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f627c1c90f0 con 0x7f627c13ee90 2024-03-29T11:51:17.118 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.110+0000 7f62797fa640 1 -- 172.21.15.98:0/3664793498 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1527285325 0 0) 0x7f627c1c68f0 con 0x7f627c0a9ac0 2024-03-29T11:51:17.118 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.110+0000 7f62797fa640 10 monclient(hunting): _negotiate 2024-03-29T11:51:17.118 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.110+0000 7f62797fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:17.118 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.110+0000 7f62797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:17.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.110+0000 7f62797fa640 10 monclient(hunting): my global_id is 4334 2024-03-29T11:51:17.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.110+0000 7f62797fa640 1 -- 172.21.15.98:0/3664793498 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f627c1c74e0 con 0x7f627c0a9ac0 2024-03-29T11:51:17.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.110+0000 7f62797fa640 1 -- 172.21.15.98:0/3664793498 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 345846798 0 0) 0x7f627c1c74e0 con 0x7f627c0a9ac0 2024-03-29T11:51:17.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.110+0000 7f62797fa640 1 -- 172.21.15.98:0/3664793498 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f627c1c68f0 con 0x7f627c0a9ac0 2024-03-29T11:51:17.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.110+0000 7f62797fa640 1 -- 172.21.15.98:0/3664793498 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f626c001640 con 0x7f627c0a9ac0 2024-03-29T11:51:17.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.110+0000 7f62797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:17.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.110+0000 7f62797fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:51:17.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.110+0000 7f62797fa640 10 monclient(hunting): dump: 2024-03-29T11:51:17.119 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:51:17.119 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:17.119 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:17.119 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:17.119 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:17.119 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:17.119 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:51:17.120 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:51:17.120 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:51:17.120 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:17.120 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.110+0000 7f62797fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:51:17.120 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.110+0000 7f62797fa640 1 -- 172.21.15.98:0/3664793498 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2549679407 0 0) 0x7f6268004eb0 con 0x7f6264004390 2024-03-29T11:51:17.120 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.110+0000 7f62797fa640 1 -- 172.21.15.98:0/3664793498 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f627c1c74e0 con 0x7f6264004390 2024-03-29T11:51:17.120 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.110+0000 7f62797fa640 1 -- 172.21.15.98:0/3664793498 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 130809822 0 0) 0x7f627c1c90f0 con 0x7f627c13ee90 2024-03-29T11:51:17.120 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.110+0000 7f62797fa640 1 -- 172.21.15.98:0/3664793498 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6268004eb0 con 0x7f627c13ee90 2024-03-29T11:51:17.120 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.110+0000 7f62797fa640 1 -- 172.21.15.98:0/3664793498 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f6274001b00 con 0x7f6264004390 2024-03-29T11:51:17.120 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.111+0000 7f62797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:17.120 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.111+0000 7f62797fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:51:17.120 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.111+0000 7f62797fa640 10 monclient(hunting): dump: 2024-03-29T11:51:17.120 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:51:17.120 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:17.121 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:17.121 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:17.121 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:17.121 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:17.121 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:51:17.121 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:51:17.121 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:51:17.121 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:17.121 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.111+0000 7f62797fa640 1 -- 172.21.15.98:0/3664793498 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f6270001510 con 0x7f627c13ee90 2024-03-29T11:51:17.121 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.111+0000 7f62797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:17.125 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.111+0000 7f62797fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:51:17.125 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.111+0000 7f62797fa640 10 monclient(hunting): dump: 2024-03-29T11:51:17.125 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:51:17.125 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:17.125 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:17.125 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:17.126 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:17.126 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:17.126 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:51:17.126 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:51:17.126 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:51:17.126 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:17.126 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.111+0000 7f62797fa640 1 -- 172.21.15.98:0/3664793498 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1168526672 0 0) 0x7f627c1c74e0 con 0x7f6264004390 2024-03-29T11:51:17.126 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.111+0000 7f62797fa640 1 -- 172.21.15.98:0/3664793498 >> v1:172.21.15.98:6790/0 conn(0x7f627c0a9ac0 legacy=0x7f627c1c6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:17.126 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.111+0000 7f62797fa640 1 -- 172.21.15.98:0/3664793498 >> v1:172.21.15.121:6789/0 conn(0x7f627c13ee90 legacy=0x7f627c139b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:17.126 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.111+0000 7f62797fa640 10 monclient: _finish_hunting 0 2024-03-29T11:51:17.126 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.111+0000 7f62797fa640 1 monclient: found mon.a 2024-03-29T11:51:17.126 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.111+0000 7f62797fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:51:17.126 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.111+0000 7f62797fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:51:17.126 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.111+0000 7f62797fa640 1 -- 172.21.15.98:0/3664793498 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f627c13a240 con 0x7f6264004390 2024-03-29T11:51:17.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.111+0000 7f62797fa640 10 monclient: _finish_auth 0 2024-03-29T11:51:17.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.111+0000 7f62797fa640 10 monclient: _check_auth_tickets 2024-03-29T11:51:17.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.111+0000 7f62797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:51:17.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.111+0000 7f62831f4640 5 monclient: authenticate success, global_id 4323 2024-03-29T11:51:17.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.111+0000 7f62831f4640 10 monclient: _renew_subs 2024-03-29T11:51:17.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.111+0000 7f62831f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:51:17.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.111+0000 7f62831f4640 1 -- 172.21.15.98:0/3664793498 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f627c1c63a0 con 0x7f6264004390 2024-03-29T11:51:17.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.111+0000 7f62831f4640 10 client.4323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:51:17.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.111+0000 7f62831f4640 10 monclient: _renew_subs 2024-03-29T11:51:17.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.111+0000 7f62831f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:51:17.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.111+0000 7f62831f4640 1 -- 172.21.15.98:0/3664793498 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f627c1de970 con 0x7f6264004390 2024-03-29T11:51:17.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.111+0000 7f62831f4640 1 librados: init done 2024-03-29T11:51:17.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.111+0000 7f62797fa640 1 -- 172.21.15.98:0/3664793498 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6274001c60 con 0x7f6264004390 2024-03-29T11:51:17.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.111+0000 7f62797fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:51:17.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.111+0000 7f627affd640 4 set_mon_vals no callback set 2024-03-29T11:51:17.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.113+0000 7f62831f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:17.128 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.113+0000 7f62797fa640 1 -- 172.21.15.98:0/3664793498 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f6274004ea0 con 0x7f6264004390 2024-03-29T11:51:17.128 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.113+0000 7f62797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:17.128 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.113+0000 7f62797fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:51:17.128 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.113+0000 7f62797fa640 10 monclient: dump: 2024-03-29T11:51:17.128 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:51:17.128 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:17.128 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:17.128 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:17.128 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:17.128 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:17.128 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:51:17.128 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:51:17.128 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:51:17.128 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:17.128 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.113+0000 7f62831f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:17.128 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.113+0000 7f62831f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:51:17.129 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.113+0000 7f62831f4640 1 -- 172.21.15.98:0/3664793498 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f625c005080 con 0x7f6264004390 2024-03-29T11:51:17.129 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.113+0000 7f62797fa640 1 -- 172.21.15.98:0/3664793498 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f627401aeb0 con 0x7f6264004390 2024-03-29T11:51:17.129 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.114+0000 7f62797fa640 1 -- 172.21.15.98:0/3664793498 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f6274084e30 con 0x7f6264004390 2024-03-29T11:51:17.129 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.114+0000 7f62797fa640 10 client.4323.objecter ms_dispatch 0x7f627c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:51:17.129 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.114+0000 7f62797fa640 3 client.4323.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:51:17.129 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.114+0000 7f62797fa640 3 client.4323.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:51:17.129 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.114+0000 7f62797fa640 20 client.4323.objecter dump_active .. 0 homeless 2024-03-29T11:51:17.129 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.115+0000 7f627bfff640 10 monclient: get_auth_request con 0x7f626806e2b0 auth_method 0 2024-03-29T11:51:17.129 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.115+0000 7f62797fa640 10 client.4323.objecter ms_handle_connect 0x7f626806e2b0 2024-03-29T11:51:17.129 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.123+0000 7f62797fa640 1 -- 172.21.15.98:0/3664793498 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f62740507f0 con 0x7f6264004390 2024-03-29T11:51:17.130 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.123+0000 7f62797fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:17.130 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.123+0000 7f62797fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:51:17.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.459+0000 7f62831f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:17.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.459+0000 7f62831f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:17.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.459+0000 7f62831f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:51:17.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.459+0000 7f62831f4640 1 -- 172.21.15.98:0/3664793498 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f625c005330 con 0x7f6264004390 2024-03-29T11:51:17.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.460+0000 7f62797fa640 1 -- 172.21.15.98:0/3664793498 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f6274052720 con 0x7f6264004390 2024-03-29T11:51:17.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.460+0000 7f62797fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:17.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.460+0000 7f62797fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:51:17.467 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:51:17.467 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:51:17.471 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.464+0000 7f62831f4640 10 librados: watch_flush enter 2024-03-29T11:51:17.471 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.464+0000 7f62831f4640 10 librados: watch_flush exit 2024-03-29T11:51:17.471 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.464+0000 7f62831f4640 20 client.4323.objecter shutdown clearing up homeless session... 2024-03-29T11:51:17.471 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.464+0000 7f62831f4640 10 client.4323.objecter successfully canceled tick 2024-03-29T11:51:17.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.465+0000 7f62831f4640 1 -- 172.21.15.98:0/3664793498 >> v1:172.21.15.98:6816/4065186974 conn(0x7f626806e2b0 legacy=0x7f6268070770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:17.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.465+0000 7f62831f4640 10 monclient: shutdown 2024-03-29T11:51:17.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.465+0000 7f62831f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:51:17.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.465+0000 7f62831f4640 1 -- 172.21.15.98:0/3664793498 >> v1:172.21.15.98:6789/0 conn(0x7f6264004390 legacy=0x7f627c071b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:17.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.465+0000 7f62831f4640 1 -- 172.21.15.98:0/3664793498 shutdown_connections 2024-03-29T11:51:17.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.465+0000 7f62831f4640 1 -- 172.21.15.98:0/3664793498 >> 172.21.15.98:0/3664793498 conn(0x7f627c0b35e0 msgr2=0x7f627c13d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:51:17.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.466+0000 7f62831f4640 1 -- 172.21.15.98:0/3664793498 shutdown_connections 2024-03-29T11:51:17.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.469+0000 7f62831f4640 1 -- 172.21.15.98:0/3664793498 wait complete. 2024-03-29T11:51:17.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:17.469+0000 7f62831f4640 1 librados: shutdown 2024-03-29T11:51:17.493 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:51:24.299 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:51:24.449 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.442+0000 7f252eb38640 10 monclient: get_monmap_and_config 2024-03-29T11:51:24.449 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.442+0000 7f252eb38640 10 monclient: build_initial_monmap 2024-03-29T11:51:24.449 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.442+0000 7f252eb38640 10 monclient: monmap: 2024-03-29T11:51:24.449 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:51:24.449 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:24.449 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:51:24.443216+0000 2024-03-29T11:51:24.449 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:51:24.443216+0000 2024-03-29T11:51:24.449 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:51:24.449 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:24.449 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:51:24.449 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:51:24.450 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:51:24.450 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:24.450 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.443+0000 7f252eb38640 1 Processor -- start 2024-03-29T11:51:24.450 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.444+0000 7f252eb38640 1 -- start start 2024-03-29T11:51:24.450 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.444+0000 7f252eb38640 10 monclient: init 2024-03-29T11:51:24.451 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.445+0000 7f252eb38640 10 monclient: _reopen_session rank -1 2024-03-29T11:51:24.451 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.445+0000 7f252eb38640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:51:24.452 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.445+0000 7f252eb38640 10 monclient(hunting): picked mon.noname-c con 0x7f25281457f0 addr v1:172.21.15.121:6789/0 2024-03-29T11:51:24.452 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.445+0000 7f252eb38640 10 monclient(hunting): picked mon.noname-a con 0x7f252813f650 addr v1:172.21.15.98:6789/0 2024-03-29T11:51:24.452 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.445+0000 7f252eb38640 10 monclient(hunting): picked mon.noname-b con 0x7f25280a9d70 addr v1:172.21.15.98:6790/0 2024-03-29T11:51:24.452 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.445+0000 7f252eb38640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25280aaef0 con 0x7f252813f650 2024-03-29T11:51:24.452 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.445+0000 7f252eb38640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2528145bd0 con 0x7f25281457f0 2024-03-29T11:51:24.452 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.445+0000 7f252eb38640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2528145eb0 con 0x7f25280a9d70 2024-03-29T11:51:24.453 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.445+0000 7f252eb38640 10 monclient(hunting): _renew_subs 2024-03-29T11:51:24.453 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.445+0000 7f252eb38640 10 monclient(hunting): authenticate will time out at 991.531029s 2024-03-29T11:51:24.457 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.450+0000 7f252d0d5640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f25280a9d70 0x7f252814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:38225/0 (socket says 172.21.15.98:38225) 2024-03-29T11:51:24.457 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.450+0000 7f252d0d5640 1 -- 172.21.15.98:0/1040067308 learned_addr learned my addr 172.21.15.98:0/1040067308 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:51:24.458 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.451+0000 7f25277fe640 1 -- 172.21.15.98:0/1040067308 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3625475839 0 0) 0x7f2528145eb0 con 0x7f25280a9d70 2024-03-29T11:51:24.458 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.451+0000 7f25277fe640 10 monclient(hunting): _negotiate 2024-03-29T11:51:24.458 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.451+0000 7f25277fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:24.458 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.451+0000 7f25277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:24.458 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.451+0000 7f25277fe640 10 monclient(hunting): my global_id is 4337 2024-03-29T11:51:24.458 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.451+0000 7f25277fe640 1 -- 172.21.15.98:0/1040067308 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2518003520 con 0x7f25280a9d70 2024-03-29T11:51:24.461 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.452+0000 7f25277fe640 1 -- 172.21.15.98:0/1040067308 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 493551246 0 0) 0x7f2518003520 con 0x7f25280a9d70 2024-03-29T11:51:24.461 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.452+0000 7f25277fe640 1 -- 172.21.15.98:0/1040067308 >> v1:172.21.15.121:6789/0 conn(0x7f25281457f0 legacy=0x7f252813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:51:24.461 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.452+0000 7f25277fe640 1 -- 172.21.15.98:0/1040067308 >> v1:172.21.15.98:6789/0 conn(0x7f252813f650 legacy=0x7f25280b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:51:24.461 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.452+0000 7f25277fe640 10 monclient: _finish_hunting 0 2024-03-29T11:51:24.461 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.452+0000 7f25277fe640 1 monclient: found mon.noname-b 2024-03-29T11:51:24.461 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.452+0000 7f25277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:51:24.461 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.452+0000 7f25277fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:51:24.462 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.452+0000 7f25277fe640 1 -- 172.21.15.98:0/1040067308 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2528146340 con 0x7f25280a9d70 2024-03-29T11:51:24.462 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.452+0000 7f25277fe640 10 monclient: _finish_auth 0 2024-03-29T11:51:24.462 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.452+0000 7f25277fe640 10 monclient: _check_auth_tickets 2024-03-29T11:51:24.462 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.452+0000 7f25277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:51:24.462 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.452+0000 7f25277fe640 1 -- 172.21.15.98:0/1040067308 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f2520001a10 con 0x7f25280a9d70 2024-03-29T11:51:24.462 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.452+0000 7f25277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:24.462 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.452+0000 7f25277fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:51:24.462 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.452+0000 7f25277fe640 10 monclient: dump: 2024-03-29T11:51:24.462 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:51:24.462 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:24.462 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:24.462 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:24.462 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:24.462 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:24.463 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:51:24.463 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:51:24.463 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:51:24.463 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:24.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.452+0000 7f25277fe640 1 -- 172.21.15.98:0/1040067308 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2520001ca0 con 0x7f25280a9d70 2024-03-29T11:51:24.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.452+0000 7f25277fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:51:24.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.452+0000 7f25277fe640 1 -- 172.21.15.98:0/1040067308 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f2520002030 con 0x7f25280a9d70 2024-03-29T11:51:24.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.452+0000 7f25277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:24.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.452+0000 7f25277fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:51:24.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.452+0000 7f25277fe640 10 monclient: dump: 2024-03-29T11:51:24.463 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:51:24.463 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:24.463 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:24.463 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:24.464 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:24.464 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:24.464 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:51:24.464 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:51:24.464 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:51:24.464 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:24.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.452+0000 7f252eb38640 5 monclient: authenticate success, global_id 4337 2024-03-29T11:51:24.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.452+0000 7f252eb38640 10 monclient: get_monmap_and_config success 2024-03-29T11:51:24.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.452+0000 7f252eb38640 4 set_mon_vals no callback set 2024-03-29T11:51:24.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.453+0000 7f252eb38640 10 monclient: shutdown 2024-03-29T11:51:24.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.453+0000 7f252eb38640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:51:24.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.453+0000 7f252eb38640 1 -- 172.21.15.98:0/1040067308 >> v1:172.21.15.98:6790/0 conn(0x7f25280a9d70 legacy=0x7f252814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:24.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.454+0000 7f252eb38640 1 -- 172.21.15.98:0/1040067308 shutdown_connections 2024-03-29T11:51:24.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.454+0000 7f252eb38640 1 -- 172.21.15.98:0/1040067308 >> 172.21.15.98:0/1040067308 conn(0x7f2528066a00 msgr2=0x7f2528066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:51:24.465 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.456+0000 7f252eb38640 1 -- 172.21.15.98:0/1040067308 shutdown_connections 2024-03-29T11:51:24.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.458+0000 7f252eb38640 1 -- 172.21.15.98:0/1040067308 wait complete. 2024-03-29T11:51:24.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.459+0000 7f252eb38640 10 monclient: build_initial_monmap 2024-03-29T11:51:24.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.459+0000 7f252eb38640 10 monclient: monmap: 2024-03-29T11:51:24.466 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:51:24.466 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:51:24.466 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:51:24.466 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:51:24.466 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:51:24.466 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:24.466 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:51:24.467 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:51:24.467 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:51:24.467 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:24.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.459+0000 7f252eb38640 1 librados: starting msgr at 2024-03-29T11:51:24.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.459+0000 7f252eb38640 1 librados: starting objecter 2024-03-29T11:51:24.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.459+0000 7f252eb38640 1 Processor -- start 2024-03-29T11:51:24.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.459+0000 7f252eb38640 1 -- start start 2024-03-29T11:51:24.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.459+0000 7f252eb38640 1 librados: setting wanted keys 2024-03-29T11:51:24.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.459+0000 7f252eb38640 1 librados: calling monclient init 2024-03-29T11:51:24.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.459+0000 7f252eb38640 10 monclient: init 2024-03-29T11:51:24.468 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.460+0000 7f252eb38640 10 monclient: _reopen_session rank -1 2024-03-29T11:51:24.468 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.460+0000 7f252eb38640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:51:24.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.464+0000 7f252eb38640 10 monclient(hunting): picked mon.noname-b con 0x7f25280a9d70 addr v1:172.21.15.121:6789/0 2024-03-29T11:51:24.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.467+0000 7f252c8d4640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f25280a9d70 0x7f2528133790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:55417/0 (socket says 172.21.15.98:55417) 2024-03-29T11:51:24.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.467+0000 7f252c8d4640 1 -- 172.21.15.98:0/2263882247 learned_addr learned my addr 172.21.15.98:0/2263882247 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:51:24.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.467+0000 7f252eb38640 10 monclient(hunting): picked mon.noname-c con 0x7f252813f650 addr v1:172.21.15.98:6790/0 2024-03-29T11:51:24.476 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.468+0000 7f252eb38640 10 monclient(hunting): picked mon.noname-a con 0x7f25281457f0 addr v1:172.21.15.98:6789/0 2024-03-29T11:51:24.476 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.468+0000 7f252eb38640 1 -- 172.21.15.98:0/2263882247 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2528136440 con 0x7f25281457f0 2024-03-29T11:51:24.476 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.468+0000 7f252eb38640 1 -- 172.21.15.98:0/2263882247 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2528135b80 con 0x7f25280a9d70 2024-03-29T11:51:24.476 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.468+0000 7f252eb38640 1 -- 172.21.15.98:0/2263882247 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2528135df0 con 0x7f252813f650 2024-03-29T11:51:24.476 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.468+0000 7f252eb38640 10 monclient(hunting): _renew_subs 2024-03-29T11:51:24.476 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.468+0000 7f252eb38640 10 monclient(hunting): authenticate will time out at 991.553287s 2024-03-29T11:51:24.476 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.469+0000 7f25257fa640 10 client.?.objecter ms_handle_connect 0x7f252813f650 2024-03-29T11:51:24.476 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.469+0000 7f25257fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:51:24.476 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.469+0000 7f25257fa640 1 -- 172.21.15.98:0/2263882247 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2180994050 0 0) 0x7f2528135df0 con 0x7f252813f650 2024-03-29T11:51:24.476 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.469+0000 7f25257fa640 10 monclient(hunting): _negotiate 2024-03-29T11:51:24.476 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.469+0000 7f25257fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:24.477 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.469+0000 7f25257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:24.477 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.469+0000 7f25257fa640 10 monclient(hunting): my global_id is 4340 2024-03-29T11:51:24.477 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.469+0000 7f25257fa640 1 -- 172.21.15.98:0/2263882247 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f250c004eb0 con 0x7f252813f650 2024-03-29T11:51:24.481 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.473+0000 7f25257fa640 10 client.?.objecter ms_handle_connect 0x7f25280a9d70 2024-03-29T11:51:24.481 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.473+0000 7f25257fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:51:24.481 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.473+0000 7f25257fa640 10 client.?.objecter ms_handle_connect 0x7f25281457f0 2024-03-29T11:51:24.481 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.473+0000 7f25257fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:51:24.481 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.473+0000 7f25257fa640 1 -- 172.21.15.98:0/2263882247 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3984178587 0 0) 0x7f2528135b80 con 0x7f25280a9d70 2024-03-29T11:51:24.481 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.473+0000 7f25257fa640 10 monclient(hunting): _negotiate 2024-03-29T11:51:24.481 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.473+0000 7f25257fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:24.481 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.473+0000 7f25257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:24.481 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.473+0000 7f25257fa640 10 monclient(hunting): my global_id is 4360 2024-03-29T11:51:24.481 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.473+0000 7f25257fa640 1 -- 172.21.15.98:0/2263882247 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2528135df0 con 0x7f25280a9d70 2024-03-29T11:51:24.481 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.473+0000 7f25257fa640 1 -- 172.21.15.98:0/2263882247 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3312728230 0 0) 0x7f250c004eb0 con 0x7f252813f650 2024-03-29T11:51:24.481 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.473+0000 7f25257fa640 1 -- 172.21.15.98:0/2263882247 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2528135b80 con 0x7f252813f650 2024-03-29T11:51:24.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.473+0000 7f25257fa640 1 -- 172.21.15.98:0/2263882247 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3163098892 0 0) 0x7f2528136440 con 0x7f25281457f0 2024-03-29T11:51:24.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.473+0000 7f25257fa640 10 monclient(hunting): _negotiate 2024-03-29T11:51:24.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.473+0000 7f25257fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:24.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.473+0000 7f25257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:24.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.473+0000 7f25257fa640 10 monclient(hunting): my global_id is 4326 2024-03-29T11:51:24.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.473+0000 7f25257fa640 1 -- 172.21.15.98:0/2263882247 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f250c004eb0 con 0x7f25281457f0 2024-03-29T11:51:24.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.473+0000 7f25257fa640 1 -- 172.21.15.98:0/2263882247 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f2514001490 con 0x7f252813f650 2024-03-29T11:51:24.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.473+0000 7f25257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:24.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.473+0000 7f25257fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:51:24.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.473+0000 7f25257fa640 10 monclient(hunting): dump: 2024-03-29T11:51:24.482 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:51:24.482 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:24.482 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:24.482 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:24.483 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:24.483 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:24.483 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:51:24.483 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:51:24.483 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:51:24.483 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:24.483 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.473+0000 7f25257fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:51:24.483 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.473+0000 7f25257fa640 1 -- 172.21.15.98:0/2263882247 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 761372337 0 0) 0x7f2528135b80 con 0x7f252813f650 2024-03-29T11:51:24.514 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.474+0000 7f25257fa640 1 -- 172.21.15.98:0/2263882247 >> v1:172.21.15.121:6789/0 conn(0x7f25280a9d70 legacy=0x7f2528133790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:24.514 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.474+0000 7f25257fa640 1 -- 172.21.15.98:0/2263882247 >> v1:172.21.15.98:6789/0 conn(0x7f25281457f0 legacy=0x7f252812efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:24.514 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.474+0000 7f25257fa640 10 monclient: _finish_hunting 0 2024-03-29T11:51:24.514 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.474+0000 7f25257fa640 1 monclient: found mon.c 2024-03-29T11:51:24.514 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.474+0000 7f25257fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:51:24.514 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.474+0000 7f25257fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:51:24.515 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.474+0000 7f25257fa640 1 -- 172.21.15.98:0/2263882247 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f252808bda0 con 0x7f252813f650 2024-03-29T11:51:24.515 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.474+0000 7f25257fa640 10 monclient: _finish_auth 0 2024-03-29T11:51:24.515 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.474+0000 7f25257fa640 10 monclient: _check_auth_tickets 2024-03-29T11:51:24.515 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.474+0000 7f25257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:51:24.515 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.474+0000 7f252eb38640 5 monclient: authenticate success, global_id 4340 2024-03-29T11:51:24.515 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.474+0000 7f252eb38640 10 monclient: _renew_subs 2024-03-29T11:51:24.515 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.474+0000 7f252eb38640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:51:24.515 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.474+0000 7f252eb38640 1 -- 172.21.15.98:0/2263882247 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f252813b4a0 con 0x7f252813f650 2024-03-29T11:51:24.515 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.474+0000 7f252eb38640 10 client.4340.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:51:24.515 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.474+0000 7f252eb38640 10 monclient: _renew_subs 2024-03-29T11:51:24.515 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.474+0000 7f252eb38640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:51:24.515 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.474+0000 7f252eb38640 1 -- 172.21.15.98:0/2263882247 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f25281424b0 con 0x7f252813f650 2024-03-29T11:51:24.515 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.474+0000 7f252eb38640 1 librados: init done 2024-03-29T11:51:24.515 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.474+0000 7f25257fa640 1 -- 172.21.15.98:0/2263882247 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25140015f0 con 0x7f252813f650 2024-03-29T11:51:24.515 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.474+0000 7f25257fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:51:24.516 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.474+0000 7f25257fa640 1 -- 172.21.15.98:0/2263882247 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f2514001c80 con 0x7f252813f650 2024-03-29T11:51:24.516 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.474+0000 7f25257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:24.516 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.474+0000 7f25257fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:51:24.516 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.474+0000 7f25257fa640 10 monclient: dump: 2024-03-29T11:51:24.516 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:51:24.516 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:24.516 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:24.516 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:24.516 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:24.516 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:24.516 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:51:24.516 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:51:24.516 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:51:24.516 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:24.517 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.475+0000 7f2526ffd640 4 set_mon_vals no callback set 2024-03-29T11:51:24.517 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.476+0000 7f25257fa640 1 -- 172.21.15.98:0/2263882247 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f2514019780 con 0x7f252813f650 2024-03-29T11:51:24.517 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.476+0000 7f2512ffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:24.517 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.476+0000 7f2512ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:24.517 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.476+0000 7f2512ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:51:24.517 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.476+0000 7f2512ffd640 1 -- 172.21.15.98:0/2263882247 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2500005080 con 0x7f252813f650 2024-03-29T11:51:24.517 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.477+0000 7f252c8d4640 10 monclient: get_auth_request con 0x7f250c06dc60 auth_method 0 2024-03-29T11:51:24.517 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.477+0000 7f25257fa640 1 -- 172.21.15.98:0/2263882247 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f2514083750 con 0x7f252813f650 2024-03-29T11:51:24.517 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.477+0000 7f25257fa640 10 client.4340.objecter ms_dispatch 0x7f2528000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:51:24.517 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.477+0000 7f25257fa640 3 client.4340.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:51:24.517 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.477+0000 7f25257fa640 3 client.4340.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:51:24.517 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.477+0000 7f25257fa640 20 client.4340.objecter dump_active .. 0 homeless 2024-03-29T11:51:24.517 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.477+0000 7f25257fa640 10 client.4340.objecter ms_handle_connect 0x7f250c06dc60 2024-03-29T11:51:24.517 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.485+0000 7f25257fa640 1 -- 172.21.15.98:0/2263882247 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f251404f030 con 0x7f252813f650 2024-03-29T11:51:24.517 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.485+0000 7f25257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:24.517 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.485+0000 7f25257fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:51:24.824 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.817+0000 7f2512ffd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:24.825 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.817+0000 7f2512ffd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:24.825 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.817+0000 7f2512ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:51:24.825 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.817+0000 7f2512ffd640 1 -- 172.21.15.98:0/2263882247 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2500005330 con 0x7f252813f650 2024-03-29T11:51:24.828 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.821+0000 7f25257fa640 1 -- 172.21.15.98:0/2263882247 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f2514050f60 con 0x7f252813f650 2024-03-29T11:51:24.828 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.821+0000 7f25257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:24.828 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.821+0000 7f25257fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:51:24.829 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:51:24.829 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:51:24.834 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.827+0000 7f2512ffd640 10 librados: watch_flush enter 2024-03-29T11:51:24.834 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.827+0000 7f2512ffd640 10 librados: watch_flush exit 2024-03-29T11:51:24.834 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.827+0000 7f2512ffd640 20 client.4340.objecter shutdown clearing up homeless session... 2024-03-29T11:51:24.834 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.827+0000 7f2512ffd640 10 client.4340.objecter successfully canceled tick 2024-03-29T11:51:24.835 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.827+0000 7f2512ffd640 1 -- 172.21.15.98:0/2263882247 >> v1:172.21.15.98:6816/4065186974 conn(0x7f250c06dc60 legacy=0x7f250c070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:24.835 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.827+0000 7f2512ffd640 10 monclient: shutdown 2024-03-29T11:51:24.835 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.827+0000 7f2512ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:51:24.835 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.827+0000 7f2512ffd640 1 -- 172.21.15.98:0/2263882247 >> v1:172.21.15.98:6790/0 conn(0x7f252813f650 legacy=0x7f252812b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:24.836 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.830+0000 7f2512ffd640 1 -- 172.21.15.98:0/2263882247 shutdown_connections 2024-03-29T11:51:24.837 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.830+0000 7f2512ffd640 1 -- 172.21.15.98:0/2263882247 >> 172.21.15.98:0/2263882247 conn(0x7f2528066a00 msgr2=0x7f25280b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:51:24.837 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.830+0000 7f2512ffd640 1 -- 172.21.15.98:0/2263882247 shutdown_connections 2024-03-29T11:51:24.837 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.830+0000 7f2512ffd640 1 -- 172.21.15.98:0/2263882247 wait complete. 2024-03-29T11:51:24.837 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:24.830+0000 7f2512ffd640 1 librados: shutdown 2024-03-29T11:51:24.853 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:51:31.661 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:51:31.823 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.815+0000 7f08312ae640 10 monclient: get_monmap_and_config 2024-03-29T11:51:31.823 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.815+0000 7f08312ae640 10 monclient: build_initial_monmap 2024-03-29T11:51:31.823 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.815+0000 7f08312ae640 10 monclient: monmap: 2024-03-29T11:51:31.823 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:51:31.823 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:31.823 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:51:31.817180+0000 2024-03-29T11:51:31.823 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:51:31.817180+0000 2024-03-29T11:51:31.824 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:51:31.824 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:31.824 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:51:31.824 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:51:31.824 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:51:31.824 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:31.825 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.816+0000 7f08312ae640 1 Processor -- start 2024-03-29T11:51:31.828 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.820+0000 7f08312ae640 1 -- start start 2024-03-29T11:51:31.828 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.820+0000 7f08312ae640 10 monclient: init 2024-03-29T11:51:31.830 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.821+0000 7f08312ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:51:31.830 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.821+0000 7f08312ae640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:51:31.830 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.821+0000 7f08312ae640 10 monclient(hunting): picked mon.noname-a con 0x7f082c0b1f60 addr v1:172.21.15.98:6789/0 2024-03-29T11:51:31.830 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.821+0000 7f08312ae640 10 monclient(hunting): picked mon.noname-c con 0x7f082c071eb0 addr v1:172.21.15.121:6789/0 2024-03-29T11:51:31.830 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.821+0000 7f08312ae640 10 monclient(hunting): picked mon.noname-b con 0x7f082c143d10 addr v1:172.21.15.98:6790/0 2024-03-29T11:51:31.830 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.821+0000 7f08312ae640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f082c06ef80 con 0x7f082c0b1f60 2024-03-29T11:51:31.830 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.821+0000 7f08312ae640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f082c06f260 con 0x7f082c071eb0 2024-03-29T11:51:31.831 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.821+0000 7f08312ae640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f082c0b40c0 con 0x7f082c143d10 2024-03-29T11:51:31.831 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.821+0000 7f08312ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:51:31.831 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.821+0000 7f08312ae640 10 monclient(hunting): authenticate will time out at 998.907398s 2024-03-29T11:51:31.832 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.824+0000 7f082a7fc640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f082c071eb0 0x7f082c146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:54718/0 (socket says 172.21.15.98:54718) 2024-03-29T11:51:31.832 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.824+0000 7f082a7fc640 1 -- 172.21.15.98:0/1135295422 learned_addr learned my addr 172.21.15.98:0/1135295422 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:51:31.832 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.825+0000 7f0829ffb640 1 -- 172.21.15.98:0/1135295422 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3322453503 0 0) 0x7f082c06f260 con 0x7f082c071eb0 2024-03-29T11:51:31.832 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.825+0000 7f0829ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:51:31.833 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.825+0000 7f0829ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:31.833 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.825+0000 7f0829ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:31.833 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.825+0000 7f0829ffb640 10 monclient(hunting): my global_id is 4363 2024-03-29T11:51:31.833 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.825+0000 7f0829ffb640 1 -- 172.21.15.98:0/1135295422 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0814003520 con 0x7f082c071eb0 2024-03-29T11:51:31.834 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.825+0000 7f0829ffb640 1 -- 172.21.15.98:0/1135295422 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1496251972 0 0) 0x7f0814003520 con 0x7f082c071eb0 2024-03-29T11:51:31.834 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.825+0000 7f0829ffb640 1 -- 172.21.15.98:0/1135295422 >> v1:172.21.15.98:6790/0 conn(0x7f082c143d10 legacy=0x7f082c146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:51:31.834 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.825+0000 7f0829ffb640 1 -- 172.21.15.98:0/1135295422 >> v1:172.21.15.98:6789/0 conn(0x7f082c0b1f60 legacy=0x7f082c143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:51:31.834 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.825+0000 7f0829ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:51:31.834 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.825+0000 7f0829ffb640 1 monclient: found mon.noname-c 2024-03-29T11:51:31.834 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.825+0000 7f0829ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:51:31.834 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.825+0000 7f0829ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:51:31.834 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.825+0000 7f0829ffb640 1 -- 172.21.15.98:0/1135295422 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f082c155d70 con 0x7f082c071eb0 2024-03-29T11:51:31.834 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.825+0000 7f0829ffb640 10 monclient: _finish_auth 0 2024-03-29T11:51:31.834 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.825+0000 7f0829ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:51:31.834 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.825+0000 7f0829ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:51:31.834 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.825+0000 7f0829ffb640 1 -- 172.21.15.98:0/1135295422 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f0824001a10 con 0x7f082c071eb0 2024-03-29T11:51:31.834 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.825+0000 7f08312ae640 5 monclient: authenticate success, global_id 4363 2024-03-29T11:51:31.835 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.825+0000 7f08312ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:51:31.835 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.825+0000 7f0829ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:31.835 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.825+0000 7f0829ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:51:31.835 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.825+0000 7f0829ffb640 10 monclient: dump: 2024-03-29T11:51:31.835 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:51:31.835 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:31.835 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:31.835 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:31.835 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:31.835 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:31.835 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:51:31.835 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:51:31.835 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:51:31.835 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:31.835 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.826+0000 7f0829ffb640 1 -- 172.21.15.98:0/1135295422 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0824001be0 con 0x7f082c071eb0 2024-03-29T11:51:31.835 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.826+0000 7f0829ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:51:31.836 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.826+0000 7f0829ffb640 1 -- 172.21.15.98:0/1135295422 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f0824002030 con 0x7f082c071eb0 2024-03-29T11:51:31.836 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.826+0000 7f0829ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:31.836 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.826+0000 7f0829ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:51:31.836 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.826+0000 7f0829ffb640 10 monclient: dump: 2024-03-29T11:51:31.836 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:51:31.836 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:31.836 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:31.836 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:31.836 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:31.836 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:31.836 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:51:31.836 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:51:31.836 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:51:31.836 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:31.837 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.826+0000 7f08312ae640 10 monclient: get_monmap_and_config success 2024-03-29T11:51:31.837 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.826+0000 7f08312ae640 4 set_mon_vals no callback set 2024-03-29T11:51:31.837 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.827+0000 7f08312ae640 10 monclient: shutdown 2024-03-29T11:51:31.837 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.827+0000 7f08312ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:51:31.837 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.827+0000 7f08312ae640 1 -- 172.21.15.98:0/1135295422 >> v1:172.21.15.121:6789/0 conn(0x7f082c071eb0 legacy=0x7f082c146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:31.838 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.830+0000 7f08312ae640 1 -- 172.21.15.98:0/1135295422 shutdown_connections 2024-03-29T11:51:31.838 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.830+0000 7f08312ae640 1 -- 172.21.15.98:0/1135295422 >> 172.21.15.98:0/1135295422 conn(0x7f082c066a00 msgr2=0x7f082c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:51:31.838 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.830+0000 7f08312ae640 1 -- 172.21.15.98:0/1135295422 shutdown_connections 2024-03-29T11:51:31.839 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.831+0000 7f08312ae640 1 -- 172.21.15.98:0/1135295422 wait complete. 2024-03-29T11:51:31.839 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.832+0000 7f08312ae640 10 monclient: build_initial_monmap 2024-03-29T11:51:31.840 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.832+0000 7f08312ae640 10 monclient: monmap: 2024-03-29T11:51:31.840 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:51:31.840 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:51:31.840 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:51:31.840 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:51:31.840 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:51:31.840 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:31.840 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:51:31.840 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:51:31.840 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:51:31.840 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:31.841 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.832+0000 7f08312ae640 1 librados: starting msgr at 2024-03-29T11:51:31.842 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.832+0000 7f08312ae640 1 librados: starting objecter 2024-03-29T11:51:31.842 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.833+0000 7f08312ae640 1 Processor -- start 2024-03-29T11:51:31.842 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.833+0000 7f08312ae640 1 -- start start 2024-03-29T11:51:31.842 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.833+0000 7f08312ae640 1 librados: setting wanted keys 2024-03-29T11:51:31.842 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.833+0000 7f08312ae640 1 librados: calling monclient init 2024-03-29T11:51:31.842 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.833+0000 7f08312ae640 10 monclient: init 2024-03-29T11:51:31.842 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.834+0000 7f08312ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:51:31.842 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.834+0000 7f08312ae640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:51:31.847 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.838+0000 7f08312ae640 10 monclient(hunting): picked mon.noname-c con 0x7f082c0b1f60 addr v1:172.21.15.98:6790/0 2024-03-29T11:51:31.847 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.838+0000 7f08312ae640 10 monclient(hunting): picked mon.noname-b con 0x7f082c143d10 addr v1:172.21.15.121:6789/0 2024-03-29T11:51:31.847 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.838+0000 7f08312ae640 10 monclient(hunting): picked mon.noname-a con 0x7f082c06fda0 addr v1:172.21.15.98:6789/0 2024-03-29T11:51:31.847 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.838+0000 7f08312ae640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f082c071770 con 0x7f082c06fda0 2024-03-29T11:51:31.847 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.838+0000 7f08312ae640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f082c070880 con 0x7f082c143d10 2024-03-29T11:51:31.847 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.838+0000 7f08312ae640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f082c070b60 con 0x7f082c0b1f60 2024-03-29T11:51:31.847 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.838+0000 7f082a7fc640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f082c143d10 0x7f082c1def60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:57320/0 (socket says 172.21.15.98:57320) 2024-03-29T11:51:31.847 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.838+0000 7f082a7fc640 1 -- 172.21.15.98:0/3946256677 learned_addr learned my addr 172.21.15.98:0/3946256677 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:51:31.847 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.838+0000 7f08312ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:51:31.847 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.838+0000 7f08312ae640 10 monclient(hunting): authenticate will time out at 998.924679s 2024-03-29T11:51:31.847 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.838+0000 7f08137fe640 10 client.?.objecter ms_handle_connect 0x7f082c143d10 2024-03-29T11:51:31.847 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.838+0000 7f08137fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:51:31.847 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.839+0000 7f08137fe640 10 client.?.objecter ms_handle_connect 0x7f082c0b1f60 2024-03-29T11:51:31.848 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.839+0000 7f08137fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:51:31.850 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.843+0000 7f08137fe640 10 client.?.objecter ms_handle_connect 0x7f082c06fda0 2024-03-29T11:51:31.850 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.843+0000 7f08137fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:51:31.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.844+0000 7f08137fe640 1 -- 172.21.15.98:0/3946256677 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 985747364 0 0) 0x7f082c070880 con 0x7f082c143d10 2024-03-29T11:51:31.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.844+0000 7f08137fe640 10 monclient(hunting): _negotiate 2024-03-29T11:51:31.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.844+0000 7f08137fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:31.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.844+0000 7f08137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:31.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.844+0000 7f08137fe640 10 monclient(hunting): my global_id is 4366 2024-03-29T11:51:31.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.844+0000 7f08137fe640 1 -- 172.21.15.98:0/3946256677 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0818004eb0 con 0x7f082c143d10 2024-03-29T11:51:31.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.844+0000 7f08137fe640 1 -- 172.21.15.98:0/3946256677 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2616575598 0 0) 0x7f082c071770 con 0x7f082c06fda0 2024-03-29T11:51:31.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.844+0000 7f08137fe640 10 monclient(hunting): _negotiate 2024-03-29T11:51:31.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.844+0000 7f08137fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:31.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.844+0000 7f08137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:31.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.844+0000 7f08137fe640 10 monclient(hunting): my global_id is 4329 2024-03-29T11:51:31.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.844+0000 7f08137fe640 1 -- 172.21.15.98:0/3946256677 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f082c070880 con 0x7f082c06fda0 2024-03-29T11:51:31.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.844+0000 7f08137fe640 1 -- 172.21.15.98:0/3946256677 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 499250469 0 0) 0x7f082c070b60 con 0x7f082c0b1f60 2024-03-29T11:51:31.854 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.844+0000 7f08137fe640 10 monclient(hunting): _negotiate 2024-03-29T11:51:31.854 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.844+0000 7f08137fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:31.854 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.844+0000 7f08137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:31.854 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.844+0000 7f08137fe640 10 monclient(hunting): my global_id is 4343 2024-03-29T11:51:31.854 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.845+0000 7f08137fe640 1 -- 172.21.15.98:0/3946256677 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f082c071770 con 0x7f082c0b1f60 2024-03-29T11:51:31.854 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.845+0000 7f08137fe640 1 -- 172.21.15.98:0/3946256677 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3762787320 0 0) 0x7f0818004eb0 con 0x7f082c143d10 2024-03-29T11:51:31.854 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.845+0000 7f08137fe640 1 -- 172.21.15.98:0/3946256677 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f082c070b60 con 0x7f082c143d10 2024-03-29T11:51:31.855 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.845+0000 7f08137fe640 1 -- 172.21.15.98:0/3946256677 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1832318323 0 0) 0x7f082c070880 con 0x7f082c06fda0 2024-03-29T11:51:31.855 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.845+0000 7f08137fe640 1 -- 172.21.15.98:0/3946256677 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0818004eb0 con 0x7f082c06fda0 2024-03-29T11:51:31.855 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.845+0000 7f08137fe640 1 -- 172.21.15.98:0/3946256677 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3593325879 0 0) 0x7f082c071770 con 0x7f082c0b1f60 2024-03-29T11:51:31.855 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.845+0000 7f08137fe640 1 -- 172.21.15.98:0/3946256677 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f082c070880 con 0x7f082c0b1f60 2024-03-29T11:51:31.855 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.845+0000 7f08137fe640 1 -- 172.21.15.98:0/3946256677 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f0824001ef0 con 0x7f082c143d10 2024-03-29T11:51:31.855 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.845+0000 7f08137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:31.855 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.845+0000 7f08137fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:51:31.855 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.845+0000 7f08137fe640 10 monclient(hunting): dump: 2024-03-29T11:51:31.855 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:51:31.855 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:31.855 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:31.856 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:31.856 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:31.856 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:31.856 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:51:31.856 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:51:31.856 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:51:31.856 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:31.856 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.845+0000 7f08137fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:51:31.856 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.845+0000 7f08137fe640 1 -- 172.21.15.98:0/3946256677 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f0820001670 con 0x7f082c06fda0 2024-03-29T11:51:31.856 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.845+0000 7f08137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:31.856 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.845+0000 7f08137fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:51:31.856 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.845+0000 7f08137fe640 10 monclient(hunting): dump: 2024-03-29T11:51:31.856 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:51:31.856 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:31.857 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:31.857 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:31.857 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:31.857 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:31.857 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:51:31.857 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:51:31.857 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:51:31.857 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:31.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.845+0000 7f08137fe640 1 -- 172.21.15.98:0/3946256677 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f081c001750 con 0x7f082c0b1f60 2024-03-29T11:51:31.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.845+0000 7f08137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:31.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.845+0000 7f08137fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:51:31.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.845+0000 7f08137fe640 10 monclient(hunting): dump: 2024-03-29T11:51:31.857 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:51:31.858 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:31.858 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:31.858 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:31.858 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:31.858 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:31.858 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:51:31.858 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:51:31.858 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:51:31.858 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:31.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.849+0000 7f08137fe640 1 -- 172.21.15.98:0/3946256677 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 96314517 0 0) 0x7f082c070b60 con 0x7f082c143d10 2024-03-29T11:51:31.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.849+0000 7f08137fe640 1 -- 172.21.15.98:0/3946256677 >> v1:172.21.15.98:6790/0 conn(0x7f082c0b1f60 legacy=0x7f082c06f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:31.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.849+0000 7f08137fe640 1 -- 172.21.15.98:0/3946256677 >> v1:172.21.15.98:6789/0 conn(0x7f082c06fda0 legacy=0x7f082c1e6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:31.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.849+0000 7f08137fe640 10 monclient: _finish_hunting 0 2024-03-29T11:51:31.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.849+0000 7f08137fe640 1 monclient: found mon.b 2024-03-29T11:51:31.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.849+0000 7f08137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:51:31.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.849+0000 7f08137fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:51:31.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.849+0000 7f08137fe640 1 -- 172.21.15.98:0/3946256677 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f082c070e40 con 0x7f082c143d10 2024-03-29T11:51:31.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.849+0000 7f08137fe640 10 monclient: _finish_auth 0 2024-03-29T11:51:31.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.849+0000 7f08137fe640 10 monclient: _check_auth_tickets 2024-03-29T11:51:31.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.849+0000 7f08137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:51:31.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.849+0000 7f08312ae640 5 monclient: authenticate success, global_id 4366 2024-03-29T11:51:31.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.849+0000 7f08312ae640 10 monclient: _renew_subs 2024-03-29T11:51:31.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.849+0000 7f08312ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:51:31.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.849+0000 7f08312ae640 1 -- 172.21.15.98:0/3946256677 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f082c155560 con 0x7f082c143d10 2024-03-29T11:51:31.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.849+0000 7f08312ae640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:51:31.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.850+0000 7f08312ae640 10 monclient: _renew_subs 2024-03-29T11:51:31.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.850+0000 7f08312ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:51:31.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.850+0000 7f08312ae640 1 -- 172.21.15.98:0/3946256677 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f082c1425a0 con 0x7f082c143d10 2024-03-29T11:51:31.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.850+0000 7f08312ae640 1 librados: init done 2024-03-29T11:51:31.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.850+0000 7f08137fe640 1 -- 172.21.15.98:0/3946256677 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0824004f70 con 0x7f082c143d10 2024-03-29T11:51:31.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.850+0000 7f08137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:51:31.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.850+0000 7f08137fe640 1 -- 172.21.15.98:0/3946256677 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f0824005280 con 0x7f082c143d10 2024-03-29T11:51:31.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.850+0000 7f08297fa640 4 set_mon_vals no callback set 2024-03-29T11:51:31.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.850+0000 7f08137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:31.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.850+0000 7f08137fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:51:31.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.850+0000 7f08137fe640 10 monclient: dump: 2024-03-29T11:51:31.863 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:51:31.863 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:31.863 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:31.863 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:31.863 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:31.863 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:31.863 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:51:31.863 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:51:31.864 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:51:31.864 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:31.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.851+0000 7f08312ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:31.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.851+0000 7f08312ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:31.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.851+0000 7f08312ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:51:31.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.851+0000 7f08312ae640 1 -- 172.21.15.98:0/3946256677 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f080c005080 con 0x7f082c143d10 2024-03-29T11:51:31.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.852+0000 7f08137fe640 1 -- 172.21.15.98:0/3946256677 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f082401b250 con 0x7f082c143d10 2024-03-29T11:51:31.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.853+0000 7f08137fe640 1 -- 172.21.15.98:0/3946256677 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f0824084f60 con 0x7f082c143d10 2024-03-29T11:51:31.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.853+0000 7f08137fe640 10 client.4366.objecter ms_dispatch 0x7f082c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:51:31.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.853+0000 7f08137fe640 3 client.4366.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:51:31.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.853+0000 7f08137fe640 3 client.4366.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:51:31.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.853+0000 7f08137fe640 20 client.4366.objecter dump_active .. 0 homeless 2024-03-29T11:51:31.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.854+0000 7f082affd640 10 monclient: get_auth_request con 0x7f081806e1f0 auth_method 0 2024-03-29T11:51:31.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.854+0000 7f08137fe640 10 client.4366.objecter ms_handle_connect 0x7f081806e1f0 2024-03-29T11:51:31.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.860+0000 7f08137fe640 1 -- 172.21.15.98:0/3946256677 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0824050840 con 0x7f082c143d10 2024-03-29T11:51:31.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.860+0000 7f08137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:31.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:31.860+0000 7f08137fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:51:32.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:32.232+0000 7f08312ae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:32.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:32.232+0000 7f08312ae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:32.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:32.232+0000 7f08312ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:51:32.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:32.232+0000 7f08312ae640 1 -- 172.21.15.98:0/3946256677 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f080c005330 con 0x7f082c143d10 2024-03-29T11:51:32.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:32.232+0000 7f08137fe640 1 -- 172.21.15.98:0/3946256677 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f0824052770 con 0x7f082c143d10 2024-03-29T11:51:32.241 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:32.232+0000 7f08137fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:32.241 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:51:32.241 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:51:32.241 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:32.232+0000 7f08137fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:51:32.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:32.236+0000 7f08312ae640 10 librados: watch_flush enter 2024-03-29T11:51:32.249 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:32.240+0000 7f08312ae640 10 librados: watch_flush exit 2024-03-29T11:51:32.249 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:32.240+0000 7f08312ae640 20 client.4366.objecter shutdown clearing up homeless session... 2024-03-29T11:51:32.249 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:32.240+0000 7f08312ae640 10 client.4366.objecter successfully canceled tick 2024-03-29T11:51:32.249 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:32.241+0000 7f08312ae640 1 -- 172.21.15.98:0/3946256677 >> v1:172.21.15.98:6816/4065186974 conn(0x7f081806e1f0 legacy=0x7f08180706b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:32.249 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:32.241+0000 7f08312ae640 10 monclient: shutdown 2024-03-29T11:51:32.249 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:32.241+0000 7f08312ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:51:32.249 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:32.241+0000 7f08312ae640 1 -- 172.21.15.98:0/3946256677 >> v1:172.21.15.121:6789/0 conn(0x7f082c143d10 legacy=0x7f082c1def60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:32.249 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:32.241+0000 7f08312ae640 1 -- 172.21.15.98:0/3946256677 shutdown_connections 2024-03-29T11:51:32.249 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:32.241+0000 7f08312ae640 1 -- 172.21.15.98:0/3946256677 >> 172.21.15.98:0/3946256677 conn(0x7f082c066a00 msgr2=0x7f082c0b7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:51:32.249 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:32.242+0000 7f08312ae640 1 -- 172.21.15.98:0/3946256677 shutdown_connections 2024-03-29T11:51:32.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:32.243+0000 7f08312ae640 1 -- 172.21.15.98:0/3946256677 wait complete. 2024-03-29T11:51:32.251 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:32.243+0000 7f08312ae640 1 librados: shutdown 2024-03-29T11:51:32.268 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:51:39.075 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:51:39.221 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.214+0000 7f1c33f15640 10 monclient: get_monmap_and_config 2024-03-29T11:51:39.221 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.214+0000 7f1c33f15640 10 monclient: build_initial_monmap 2024-03-29T11:51:39.221 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.214+0000 7f1c33f15640 10 monclient: monmap: 2024-03-29T11:51:39.221 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:51:39.221 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:39.221 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:51:39.215363+0000 2024-03-29T11:51:39.221 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:51:39.215363+0000 2024-03-29T11:51:39.221 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:51:39.221 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:39.222 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:51:39.222 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:51:39.222 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:51:39.222 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:39.222 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.215+0000 7f1c33f15640 1 Processor -- start 2024-03-29T11:51:39.222 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.215+0000 7f1c33f15640 1 -- start start 2024-03-29T11:51:39.222 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.215+0000 7f1c33f15640 10 monclient: init 2024-03-29T11:51:39.223 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.216+0000 7f1c33f15640 10 monclient: _reopen_session rank -1 2024-03-29T11:51:39.223 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.216+0000 7f1c33f15640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:51:39.223 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.216+0000 7f1c33f15640 10 monclient(hunting): picked mon.noname-c con 0x7f1c2c1457f0 addr v1:172.21.15.121:6789/0 2024-03-29T11:51:39.223 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.216+0000 7f1c33f15640 10 monclient(hunting): picked mon.noname-a con 0x7f1c2c13f650 addr v1:172.21.15.98:6789/0 2024-03-29T11:51:39.223 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.216+0000 7f1c33f15640 10 monclient(hunting): picked mon.noname-b con 0x7f1c2c0a9d70 addr v1:172.21.15.98:6790/0 2024-03-29T11:51:39.223 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.216+0000 7f1c33f15640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c2c0aaef0 con 0x7f1c2c13f650 2024-03-29T11:51:39.223 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.216+0000 7f1c33f15640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c2c145bd0 con 0x7f1c2c1457f0 2024-03-29T11:51:39.223 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.216+0000 7f1c33f15640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c2c145eb0 con 0x7f1c2c0a9d70 2024-03-29T11:51:39.224 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.216+0000 7f1c33f15640 10 monclient(hunting): _renew_subs 2024-03-29T11:51:39.224 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.216+0000 7f1c33f15640 10 monclient(hunting): authenticate will time out at 1006.301686s 2024-03-29T11:51:39.224 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.216+0000 7f1c324b2640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f1c2c0a9d70 0x7f1c2c14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:46730/0 (socket says 172.21.15.98:46730) 2024-03-29T11:51:39.224 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.216+0000 7f1c324b2640 1 -- 172.21.15.98:0/2137741206 learned_addr learned my addr 172.21.15.98:0/2137741206 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:51:39.225 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.217+0000 7f1c30caf640 1 -- 172.21.15.98:0/2137741206 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3113496890 0 0) 0x7f1c2c0aaef0 con 0x7f1c2c13f650 2024-03-29T11:51:39.225 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.217+0000 7f1c30caf640 10 monclient(hunting): _negotiate 2024-03-29T11:51:39.225 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.217+0000 7f1c30caf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:39.225 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.217+0000 7f1c30caf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:39.225 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.217+0000 7f1c30caf640 10 monclient(hunting): my global_id is 4332 2024-03-29T11:51:39.226 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.217+0000 7f1c30caf640 1 -- 172.21.15.98:0/2137741206 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c18003520 con 0x7f1c2c13f650 2024-03-29T11:51:39.226 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.217+0000 7f1c30caf640 1 -- 172.21.15.98:0/2137741206 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1677760166 0 0) 0x7f1c2c145eb0 con 0x7f1c2c0a9d70 2024-03-29T11:51:39.226 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.217+0000 7f1c30caf640 10 monclient(hunting): _negotiate 2024-03-29T11:51:39.226 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.217+0000 7f1c30caf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:39.226 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.217+0000 7f1c30caf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:39.226 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.217+0000 7f1c30caf640 10 monclient(hunting): my global_id is 4346 2024-03-29T11:51:39.226 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.217+0000 7f1c30caf640 1 -- 172.21.15.98:0/2137741206 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c2c0aaef0 con 0x7f1c2c0a9d70 2024-03-29T11:51:39.226 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.217+0000 7f1c30caf640 1 -- 172.21.15.98:0/2137741206 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1785711506 0 0) 0x7f1c18003520 con 0x7f1c2c13f650 2024-03-29T11:51:39.226 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.217+0000 7f1c30caf640 1 -- 172.21.15.98:0/2137741206 >> v1:172.21.15.98:6790/0 conn(0x7f1c2c0a9d70 legacy=0x7f1c2c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:39.226 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.217+0000 7f1c30caf640 1 -- 172.21.15.98:0/2137741206 >> v1:172.21.15.121:6789/0 conn(0x7f1c2c1457f0 legacy=0x7f1c2c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:51:39.226 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.217+0000 7f1c30caf640 10 monclient: _finish_hunting 0 2024-03-29T11:51:39.226 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.217+0000 7f1c30caf640 1 monclient: found mon.noname-a 2024-03-29T11:51:39.226 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.217+0000 7f1c30caf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:51:39.226 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.217+0000 7f1c30caf640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T11:51:39.226 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.217+0000 7f1c30caf640 1 -- 172.21.15.98:0/2137741206 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c2c146340 con 0x7f1c2c13f650 2024-03-29T11:51:39.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.217+0000 7f1c30caf640 10 monclient: _finish_auth 0 2024-03-29T11:51:39.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.217+0000 7f1c30caf640 10 monclient: _check_auth_tickets 2024-03-29T11:51:39.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.217+0000 7f1c30caf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:51:39.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.217+0000 7f1c30caf640 1 -- 172.21.15.98:0/2137741206 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f1c20001690 con 0x7f1c2c13f650 2024-03-29T11:51:39.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.217+0000 7f1c30caf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:39.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.218+0000 7f1c30caf640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:51:39.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.218+0000 7f1c30caf640 10 monclient: dump: 2024-03-29T11:51:39.227 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:51:39.227 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:39.227 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:39.227 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:39.227 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:39.227 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:39.227 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:51:39.227 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:51:39.227 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:51:39.228 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:39.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.218+0000 7f1c30caf640 1 -- 172.21.15.98:0/2137741206 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c200018b0 con 0x7f1c2c13f650 2024-03-29T11:51:39.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.218+0000 7f1c30caf640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:51:39.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.218+0000 7f1c33f15640 5 monclient: authenticate success, global_id 4332 2024-03-29T11:51:39.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.218+0000 7f1c33f15640 10 monclient: get_monmap_and_config success 2024-03-29T11:51:39.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.218+0000 7f1c33f15640 4 set_mon_vals no callback set 2024-03-29T11:51:39.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.218+0000 7f1c30caf640 1 -- 172.21.15.98:0/2137741206 <== mon.0 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f1c20001c40 con 0x7f1c2c13f650 2024-03-29T11:51:39.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.218+0000 7f1c30caf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:39.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.218+0000 7f1c30caf640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:51:39.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.218+0000 7f1c30caf640 10 monclient: dump: 2024-03-29T11:51:39.228 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:51:39.228 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:39.228 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:39.228 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:39.228 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:39.229 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:39.229 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:51:39.229 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:51:39.229 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:51:39.229 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:39.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.219+0000 7f1c33f15640 10 monclient: shutdown 2024-03-29T11:51:39.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.219+0000 7f1c33f15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:51:39.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.219+0000 7f1c33f15640 1 -- 172.21.15.98:0/2137741206 >> v1:172.21.15.98:6789/0 conn(0x7f1c2c13f650 legacy=0x7f1c2c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:39.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.219+0000 7f1c33f15640 1 -- 172.21.15.98:0/2137741206 shutdown_connections 2024-03-29T11:51:39.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.219+0000 7f1c33f15640 1 -- 172.21.15.98:0/2137741206 >> 172.21.15.98:0/2137741206 conn(0x7f1c2c066a00 msgr2=0x7f1c2c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:51:39.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.219+0000 7f1c33f15640 1 -- 172.21.15.98:0/2137741206 shutdown_connections 2024-03-29T11:51:39.233 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.219+0000 7f1c33f15640 1 -- 172.21.15.98:0/2137741206 wait complete. 2024-03-29T11:51:39.233 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.220+0000 7f1c33f15640 10 monclient: build_initial_monmap 2024-03-29T11:51:39.233 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.220+0000 7f1c33f15640 10 monclient: monmap: 2024-03-29T11:51:39.233 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:51:39.233 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:51:39.233 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:51:39.233 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:51:39.233 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:51:39.233 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:39.233 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:51:39.233 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:51:39.233 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:51:39.233 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:39.233 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.220+0000 7f1c33f15640 1 librados: starting msgr at 2024-03-29T11:51:39.233 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.220+0000 7f1c33f15640 1 librados: starting objecter 2024-03-29T11:51:39.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.221+0000 7f1c33f15640 1 Processor -- start 2024-03-29T11:51:39.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.221+0000 7f1c33f15640 1 -- start start 2024-03-29T11:51:39.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.221+0000 7f1c33f15640 1 librados: setting wanted keys 2024-03-29T11:51:39.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.221+0000 7f1c33f15640 1 librados: calling monclient init 2024-03-29T11:51:39.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.221+0000 7f1c33f15640 10 monclient: init 2024-03-29T11:51:39.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.221+0000 7f1c33f15640 10 monclient: _reopen_session rank -1 2024-03-29T11:51:39.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.221+0000 7f1c33f15640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:51:39.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.221+0000 7f1c33f15640 10 monclient(hunting): picked mon.noname-c con 0x7f1c2c0a9d70 addr v1:172.21.15.98:6790/0 2024-03-29T11:51:39.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.222+0000 7f1c33f15640 10 monclient(hunting): picked mon.noname-a con 0x7f1c2c13f650 addr v1:172.21.15.98:6789/0 2024-03-29T11:51:39.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.222+0000 7f1c314b0640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f1c2c13f650 0x7f1c2c1399a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:58380/0 (socket says 172.21.15.98:58380) 2024-03-29T11:51:39.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.222+0000 7f1c314b0640 1 -- 172.21.15.98:0/993024214 learned_addr learned my addr 172.21.15.98:0/993024214 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:51:39.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.222+0000 7f1c33f15640 10 monclient(hunting): picked mon.noname-b con 0x7f1c2c1457f0 addr v1:172.21.15.121:6789/0 2024-03-29T11:51:39.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.222+0000 7f1c33f15640 1 -- 172.21.15.98:0/993024214 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c2c0b0500 con 0x7f1c2c13f650 2024-03-29T11:51:39.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.222+0000 7f1c33f15640 1 -- 172.21.15.98:0/993024214 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c2c13a7c0 con 0x7f1c2c1457f0 2024-03-29T11:51:39.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.222+0000 7f1c33f15640 1 -- 172.21.15.98:0/993024214 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c2c1db390 con 0x7f1c2c0a9d70 2024-03-29T11:51:39.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.222+0000 7f1c33f15640 10 monclient(hunting): _renew_subs 2024-03-29T11:51:39.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.222+0000 7f1c1e7fc640 10 client.?.objecter ms_handle_connect 0x7f1c2c0a9d70 2024-03-29T11:51:39.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.222+0000 7f1c1e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:51:39.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.222+0000 7f1c33f15640 10 monclient(hunting): authenticate will time out at 1006.307438s 2024-03-29T11:51:39.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.222+0000 7f1c1e7fc640 10 client.?.objecter ms_handle_connect 0x7f1c2c13f650 2024-03-29T11:51:39.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.222+0000 7f1c1e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:51:39.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.222+0000 7f1c1e7fc640 10 client.?.objecter ms_handle_connect 0x7f1c2c1457f0 2024-03-29T11:51:39.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.223+0000 7f1c1e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:51:39.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.223+0000 7f1c1e7fc640 1 -- 172.21.15.98:0/993024214 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 267098287 0 0) 0x7f1c2c1db390 con 0x7f1c2c0a9d70 2024-03-29T11:51:39.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.223+0000 7f1c1e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:51:39.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.223+0000 7f1c1e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:39.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.223+0000 7f1c1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:39.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.223+0000 7f1c1e7fc640 10 monclient(hunting): my global_id is 4349 2024-03-29T11:51:39.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.223+0000 7f1c1e7fc640 1 -- 172.21.15.98:0/993024214 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c10004eb0 con 0x7f1c2c0a9d70 2024-03-29T11:51:39.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.223+0000 7f1c1e7fc640 1 -- 172.21.15.98:0/993024214 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2540471483 0 0) 0x7f1c2c0b0500 con 0x7f1c2c13f650 2024-03-29T11:51:39.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.223+0000 7f1c1e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:51:39.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.223+0000 7f1c1e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:39.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.223+0000 7f1c1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:39.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.223+0000 7f1c1e7fc640 10 monclient(hunting): my global_id is 4335 2024-03-29T11:51:39.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.223+0000 7f1c1e7fc640 1 -- 172.21.15.98:0/993024214 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c2c1db390 con 0x7f1c2c13f650 2024-03-29T11:51:39.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.223+0000 7f1c1e7fc640 1 -- 172.21.15.98:0/993024214 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3741970506 0 0) 0x7f1c2c13a7c0 con 0x7f1c2c1457f0 2024-03-29T11:51:39.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.223+0000 7f1c1e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:51:39.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.223+0000 7f1c1e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:39.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.223+0000 7f1c1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:39.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.223+0000 7f1c1e7fc640 10 monclient(hunting): my global_id is 4369 2024-03-29T11:51:39.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.223+0000 7f1c1e7fc640 1 -- 172.21.15.98:0/993024214 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c2c0b0500 con 0x7f1c2c1457f0 2024-03-29T11:51:39.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.224+0000 7f1c1e7fc640 1 -- 172.21.15.98:0/993024214 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3031005890 0 0) 0x7f1c10004eb0 con 0x7f1c2c0a9d70 2024-03-29T11:51:39.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.224+0000 7f1c1e7fc640 1 -- 172.21.15.98:0/993024214 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c2c13a7c0 con 0x7f1c2c0a9d70 2024-03-29T11:51:39.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.224+0000 7f1c1e7fc640 1 -- 172.21.15.98:0/993024214 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3376480582 0 0) 0x7f1c2c1db390 con 0x7f1c2c13f650 2024-03-29T11:51:39.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.224+0000 7f1c1e7fc640 1 -- 172.21.15.98:0/993024214 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c10004eb0 con 0x7f1c2c13f650 2024-03-29T11:51:39.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.224+0000 7f1c1e7fc640 1 -- 172.21.15.98:0/993024214 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f1c280017d0 con 0x7f1c2c0a9d70 2024-03-29T11:51:39.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.224+0000 7f1c1e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:39.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.224+0000 7f1c1e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:51:39.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.224+0000 7f1c1e7fc640 10 monclient(hunting): dump: 2024-03-29T11:51:39.237 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:51:39.237 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:39.237 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:39.237 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:39.237 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:39.237 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:39.237 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:51:39.237 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:51:39.237 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:51:39.237 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:39.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.224+0000 7f1c1e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:51:39.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.224+0000 7f1c1e7fc640 1 -- 172.21.15.98:0/993024214 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f1c200015d0 con 0x7f1c2c13f650 2024-03-29T11:51:39.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.224+0000 7f1c1e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:39.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.224+0000 7f1c1e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:51:39.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.224+0000 7f1c1e7fc640 10 monclient(hunting): dump: 2024-03-29T11:51:39.238 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:51:39.238 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:39.238 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:39.238 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:39.238 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:39.238 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:39.238 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:51:39.238 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:51:39.238 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:51:39.238 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:39.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.224+0000 7f1c1e7fc640 1 -- 172.21.15.98:0/993024214 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3106129950 0 0) 0x7f1c2c0b0500 con 0x7f1c2c1457f0 2024-03-29T11:51:39.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.224+0000 7f1c1e7fc640 1 -- 172.21.15.98:0/993024214 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c2c1db390 con 0x7f1c2c1457f0 2024-03-29T11:51:39.239 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.224+0000 7f1c1e7fc640 1 -- 172.21.15.98:0/993024214 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f1c24001c30 con 0x7f1c2c1457f0 2024-03-29T11:51:39.239 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.224+0000 7f1c1e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:39.239 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.224+0000 7f1c1e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:51:39.239 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.224+0000 7f1c1e7fc640 10 monclient(hunting): dump: 2024-03-29T11:51:39.239 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:51:39.239 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:39.239 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:39.239 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:39.239 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:39.239 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:39.239 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:51:39.239 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:51:39.239 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:51:39.239 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:39.239 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.224+0000 7f1c1e7fc640 1 -- 172.21.15.98:0/993024214 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4176344723 0 0) 0x7f1c2c13a7c0 con 0x7f1c2c0a9d70 2024-03-29T11:51:39.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.224+0000 7f1c1e7fc640 1 -- 172.21.15.98:0/993024214 >> v1:172.21.15.121:6789/0 conn(0x7f1c2c1457f0 legacy=0x7f1c2c13a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:39.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.224+0000 7f1c1e7fc640 1 -- 172.21.15.98:0/993024214 >> v1:172.21.15.98:6789/0 conn(0x7f1c2c13f650 legacy=0x7f1c2c1399a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:39.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.224+0000 7f1c1e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:51:39.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.224+0000 7f1c1e7fc640 1 monclient: found mon.c 2024-03-29T11:51:39.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.224+0000 7f1c1e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:51:39.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.224+0000 7f1c1e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:51:39.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.224+0000 7f1c1e7fc640 1 -- 172.21.15.98:0/993024214 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c2c0b0990 con 0x7f1c2c0a9d70 2024-03-29T11:51:39.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.224+0000 7f1c1e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:51:39.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.224+0000 7f1c1e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:51:39.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.224+0000 7f1c1e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:51:39.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.224+0000 7f1c33f15640 5 monclient: authenticate success, global_id 4349 2024-03-29T11:51:39.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.224+0000 7f1c33f15640 10 monclient: _renew_subs 2024-03-29T11:51:39.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.224+0000 7f1c33f15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:51:39.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.224+0000 7f1c33f15640 1 -- 172.21.15.98:0/993024214 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c2c0b0c00 con 0x7f1c2c0a9d70 2024-03-29T11:51:39.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.224+0000 7f1c33f15640 10 client.4349.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:51:39.241 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.224+0000 7f1c33f15640 10 monclient: _renew_subs 2024-03-29T11:51:39.241 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.224+0000 7f1c33f15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:51:39.241 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.224+0000 7f1c33f15640 1 -- 172.21.15.98:0/993024214 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c2c0b10b0 con 0x7f1c2c0a9d70 2024-03-29T11:51:39.241 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.225+0000 7f1c33f15640 1 librados: init done 2024-03-29T11:51:39.241 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.225+0000 7f1c1e7fc640 1 -- 172.21.15.98:0/993024214 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c28001930 con 0x7f1c2c0a9d70 2024-03-29T11:51:39.241 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.225+0000 7f1c1e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:51:39.241 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.225+0000 7f1c1e7fc640 1 -- 172.21.15.98:0/993024214 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f1c28003790 con 0x7f1c2c0a9d70 2024-03-29T11:51:39.241 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.225+0000 7f1c1ffff640 4 set_mon_vals no callback set 2024-03-29T11:51:39.241 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.225+0000 7f1c1e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:39.241 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.225+0000 7f1c1e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:51:39.241 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.225+0000 7f1c1e7fc640 10 monclient: dump: 2024-03-29T11:51:39.241 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:51:39.241 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:39.241 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:39.241 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:39.241 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:39.242 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:39.242 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:51:39.242 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:51:39.242 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:51:39.242 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:39.243 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.226+0000 7f1c33f15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:39.243 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.227+0000 7f1c33f15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:39.243 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.227+0000 7f1c33f15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:51:39.243 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.227+0000 7f1c33f15640 1 -- 172.21.15.98:0/993024214 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c14005080 con 0x7f1c2c0a9d70 2024-03-29T11:51:39.243 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.227+0000 7f1c1e7fc640 1 -- 172.21.15.98:0/993024214 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f1c28019760 con 0x7f1c2c0a9d70 2024-03-29T11:51:39.243 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.228+0000 7f1c314b0640 10 monclient: get_auth_request con 0x7f1c1006e140 auth_method 0 2024-03-29T11:51:39.243 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.228+0000 7f1c1e7fc640 1 -- 172.21.15.98:0/993024214 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f1c28083730 con 0x7f1c2c0a9d70 2024-03-29T11:51:39.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.228+0000 7f1c1e7fc640 10 client.4349.objecter ms_dispatch 0x7f1c2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:51:39.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.228+0000 7f1c1e7fc640 3 client.4349.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:51:39.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.228+0000 7f1c1e7fc640 3 client.4349.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:51:39.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.229+0000 7f1c1e7fc640 20 client.4349.objecter dump_active .. 0 homeless 2024-03-29T11:51:39.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.229+0000 7f1c1e7fc640 10 client.4349.objecter ms_handle_connect 0x7f1c1006e140 2024-03-29T11:51:39.246 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.239+0000 7f1c1e7fc640 1 -- 172.21.15.98:0/993024214 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1c2804ee40 con 0x7f1c2c0a9d70 2024-03-29T11:51:39.246 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.239+0000 7f1c1e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:39.246 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.239+0000 7f1c1e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:51:39.627 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.620+0000 7f1c33f15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:39.627 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.620+0000 7f1c33f15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:39.627 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.620+0000 7f1c33f15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:51:39.628 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.620+0000 7f1c33f15640 1 -- 172.21.15.98:0/993024214 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c14005330 con 0x7f1c2c0a9d70 2024-03-29T11:51:39.628 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.621+0000 7f1c1e7fc640 1 -- 172.21.15.98:0/993024214 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f1c28050f40 con 0x7f1c2c0a9d70 2024-03-29T11:51:39.628 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.621+0000 7f1c1e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:39.628 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.621+0000 7f1c1e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:51:39.629 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:51:39.629 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:51:39.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.625+0000 7f1c33f15640 10 librados: watch_flush enter 2024-03-29T11:51:39.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.625+0000 7f1c33f15640 10 librados: watch_flush exit 2024-03-29T11:51:39.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.625+0000 7f1c33f15640 20 client.4349.objecter shutdown clearing up homeless session... 2024-03-29T11:51:39.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.625+0000 7f1c33f15640 10 client.4349.objecter successfully canceled tick 2024-03-29T11:51:39.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.625+0000 7f1c33f15640 1 -- 172.21.15.98:0/993024214 >> v1:172.21.15.98:6816/4065186974 conn(0x7f1c1006e140 legacy=0x7f1c10070600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:39.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.625+0000 7f1c33f15640 10 monclient: shutdown 2024-03-29T11:51:39.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.625+0000 7f1c33f15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:51:39.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.625+0000 7f1c33f15640 1 -- 172.21.15.98:0/993024214 >> v1:172.21.15.98:6790/0 conn(0x7f1c2c0a9d70 legacy=0x7f1c2c1dac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:39.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.626+0000 7f1c33f15640 1 -- 172.21.15.98:0/993024214 shutdown_connections 2024-03-29T11:51:39.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.626+0000 7f1c33f15640 1 -- 172.21.15.98:0/993024214 >> 172.21.15.98:0/993024214 conn(0x7f1c2c066a00 msgr2=0x7f1c2c0b4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:51:39.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.626+0000 7f1c33f15640 1 -- 172.21.15.98:0/993024214 shutdown_connections 2024-03-29T11:51:39.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.626+0000 7f1c33f15640 1 -- 172.21.15.98:0/993024214 wait complete. 2024-03-29T11:51:39.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:39.626+0000 7f1c33f15640 1 librados: shutdown 2024-03-29T11:51:39.650 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:51:46.459 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:51:46.595 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.587+0000 7f92b85a5640 10 monclient: get_monmap_and_config 2024-03-29T11:51:46.595 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.587+0000 7f92b85a5640 10 monclient: build_initial_monmap 2024-03-29T11:51:46.595 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.587+0000 7f92b85a5640 10 monclient: monmap: 2024-03-29T11:51:46.595 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:51:46.595 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:46.595 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:51:46.588713+0000 2024-03-29T11:51:46.595 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:51:46.588713+0000 2024-03-29T11:51:46.595 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:51:46.596 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:46.596 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:51:46.596 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:51:46.596 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:51:46.596 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:46.596 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.589+0000 7f92b85a5640 1 Processor -- start 2024-03-29T11:51:46.596 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.589+0000 7f92b85a5640 1 -- start start 2024-03-29T11:51:46.597 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.589+0000 7f92b85a5640 10 monclient: init 2024-03-29T11:51:46.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.590+0000 7f92b85a5640 10 monclient: _reopen_session rank -1 2024-03-29T11:51:46.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.590+0000 7f92b85a5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:51:46.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.590+0000 7f92b85a5640 10 monclient(hunting): picked mon.noname-a con 0x7f92b01422b0 addr v1:172.21.15.98:6789/0 2024-03-29T11:51:46.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.590+0000 7f92b85a5640 10 monclient(hunting): picked mon.noname-b con 0x7f92b0142690 addr v1:172.21.15.98:6790/0 2024-03-29T11:51:46.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.590+0000 7f92b85a5640 10 monclient(hunting): picked mon.noname-c con 0x7f92b00b17d0 addr v1:172.21.15.121:6789/0 2024-03-29T11:51:46.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.590+0000 7f92b85a5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b0142c80 con 0x7f92b01422b0 2024-03-29T11:51:46.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.590+0000 7f92b85a5640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b00aae40 con 0x7f92b00b17d0 2024-03-29T11:51:46.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.590+0000 7f92b85a5640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b013fbd0 con 0x7f92b0142690 2024-03-29T11:51:46.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.590+0000 7f92b85a5640 10 monclient(hunting): _renew_subs 2024-03-29T11:51:46.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.590+0000 7f92b85a5640 10 monclient(hunting): authenticate will time out at 1013.675854s 2024-03-29T11:51:46.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.591+0000 7f92b6341640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f92b01422b0 0x7f92b013f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43880/0 (socket says 172.21.15.98:43880) 2024-03-29T11:51:46.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.591+0000 7f92b6341640 1 -- 172.21.15.98:0/1963175790 learned_addr learned my addr 172.21.15.98:0/1963175790 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:51:46.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.591+0000 7f92b533f640 1 -- 172.21.15.98:0/1963175790 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3183465523 0 0) 0x7f92b0142c80 con 0x7f92b01422b0 2024-03-29T11:51:46.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.591+0000 7f92b533f640 10 monclient(hunting): _negotiate 2024-03-29T11:51:46.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.591+0000 7f92b533f640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:46.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.591+0000 7f92b533f640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:46.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.591+0000 7f92b533f640 10 monclient(hunting): my global_id is 4338 2024-03-29T11:51:46.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.591+0000 7f92b533f640 1 -- 172.21.15.98:0/1963175790 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f929c003520 con 0x7f92b01422b0 2024-03-29T11:51:46.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.591+0000 7f92b533f640 1 -- 172.21.15.98:0/1963175790 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3959717369 0 0) 0x7f929c003520 con 0x7f92b01422b0 2024-03-29T11:51:46.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.591+0000 7f92b533f640 1 -- 172.21.15.98:0/1963175790 >> v1:172.21.15.98:6790/0 conn(0x7f92b0142690 legacy=0x7f92b00b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:51:46.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.591+0000 7f92b533f640 1 -- 172.21.15.98:0/1963175790 >> v1:172.21.15.121:6789/0 conn(0x7f92b00b17d0 legacy=0x7f92b00b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:51:46.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.593+0000 7f92b533f640 10 monclient: _finish_hunting 0 2024-03-29T11:51:46.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.593+0000 7f92b533f640 1 monclient: found mon.noname-a 2024-03-29T11:51:46.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.593+0000 7f92b533f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:51:46.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.593+0000 7f92b533f640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T11:51:46.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.593+0000 7f92b533f640 1 -- 172.21.15.98:0/1963175790 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b0151940 con 0x7f92b01422b0 2024-03-29T11:51:46.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.593+0000 7f92b533f640 10 monclient: _finish_auth 0 2024-03-29T11:51:46.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.593+0000 7f92b533f640 10 monclient: _check_auth_tickets 2024-03-29T11:51:46.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.593+0000 7f92b533f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:51:46.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.593+0000 7f92b533f640 1 -- 172.21.15.98:0/1963175790 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f92ac001690 con 0x7f92b01422b0 2024-03-29T11:51:46.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.593+0000 7f92b533f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:46.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.593+0000 7f92b533f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:51:46.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.593+0000 7f92b533f640 10 monclient: dump: 2024-03-29T11:51:46.602 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:51:46.602 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:46.602 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:46.602 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:46.602 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:46.602 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:46.602 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:51:46.602 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:51:46.602 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:51:46.602 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:46.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.593+0000 7f92b85a5640 5 monclient: authenticate success, global_id 4338 2024-03-29T11:51:46.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.593+0000 7f92b85a5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:51:46.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.593+0000 7f92b533f640 1 -- 172.21.15.98:0/1963175790 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92ac0017f0 con 0x7f92b01422b0 2024-03-29T11:51:46.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.593+0000 7f92b533f640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:51:46.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.593+0000 7f92b533f640 1 -- 172.21.15.98:0/1963175790 <== mon.0 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f92ac001c40 con 0x7f92b01422b0 2024-03-29T11:51:46.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.593+0000 7f92b533f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:46.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.593+0000 7f92b533f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:51:46.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.593+0000 7f92b533f640 10 monclient: dump: 2024-03-29T11:51:46.603 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:51:46.603 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:46.603 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:46.603 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:46.603 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:46.603 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:46.604 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:51:46.604 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:51:46.604 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:51:46.604 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:46.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.593+0000 7f92b85a5640 10 monclient: get_monmap_and_config success 2024-03-29T11:51:46.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.593+0000 7f92b85a5640 4 set_mon_vals no callback set 2024-03-29T11:51:46.605 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.594+0000 7f92b85a5640 10 monclient: shutdown 2024-03-29T11:51:46.605 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.594+0000 7f92b85a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:51:46.605 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.594+0000 7f92b85a5640 1 -- 172.21.15.98:0/1963175790 >> v1:172.21.15.98:6789/0 conn(0x7f92b01422b0 legacy=0x7f92b013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:46.606 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.595+0000 7f92b85a5640 1 -- 172.21.15.98:0/1963175790 shutdown_connections 2024-03-29T11:51:46.606 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.595+0000 7f92b85a5640 1 -- 172.21.15.98:0/1963175790 >> 172.21.15.98:0/1963175790 conn(0x7f92b0066a00 msgr2=0x7f92b0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:51:46.606 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.595+0000 7f92b85a5640 1 -- 172.21.15.98:0/1963175790 shutdown_connections 2024-03-29T11:51:46.606 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.595+0000 7f92b85a5640 1 -- 172.21.15.98:0/1963175790 wait complete. 2024-03-29T11:51:46.606 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.596+0000 7f92b85a5640 10 monclient: build_initial_monmap 2024-03-29T11:51:46.606 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.596+0000 7f92b85a5640 10 monclient: monmap: 2024-03-29T11:51:46.606 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:51:46.606 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:51:46.606 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:51:46.606 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:51:46.606 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:51:46.606 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:46.607 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:51:46.607 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:51:46.607 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:51:46.607 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:46.607 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.597+0000 7f92b85a5640 1 librados: starting msgr at 2024-03-29T11:51:46.607 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.597+0000 7f92b85a5640 1 librados: starting objecter 2024-03-29T11:51:46.607 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.597+0000 7f92b85a5640 1 Processor -- start 2024-03-29T11:51:46.607 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.597+0000 7f92b85a5640 1 -- start start 2024-03-29T11:51:46.607 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.597+0000 7f92b85a5640 1 librados: setting wanted keys 2024-03-29T11:51:46.607 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.597+0000 7f92b85a5640 1 librados: calling monclient init 2024-03-29T11:51:46.607 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.597+0000 7f92b85a5640 10 monclient: init 2024-03-29T11:51:46.607 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.598+0000 7f92b85a5640 10 monclient: _reopen_session rank -1 2024-03-29T11:51:46.607 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.598+0000 7f92b85a5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:51:46.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.599+0000 7f92b85a5640 10 monclient(hunting): picked mon.noname-b con 0x7f92b00b17d0 addr v1:172.21.15.121:6789/0 2024-03-29T11:51:46.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.599+0000 7f92b85a5640 10 monclient(hunting): picked mon.noname-c con 0x7f92b01422b0 addr v1:172.21.15.98:6790/0 2024-03-29T11:51:46.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.599+0000 7f92b85a5640 10 monclient(hunting): picked mon.noname-a con 0x7f92b0142690 addr v1:172.21.15.98:6789/0 2024-03-29T11:51:46.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.599+0000 7f92b85a5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b006eac0 con 0x7f92b0142690 2024-03-29T11:51:46.615 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.599+0000 7f92b85a5640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b006eda0 con 0x7f92b00b17d0 2024-03-29T11:51:46.615 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.599+0000 7f92b85a5640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b0132320 con 0x7f92b01422b0 2024-03-29T11:51:46.615 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.599+0000 7f92b85a5640 10 monclient(hunting): _renew_subs 2024-03-29T11:51:46.615 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.599+0000 7f92b6b42640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f92b0142690 0x7f92b01e6fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44112/0 (socket says 172.21.15.98:44112) 2024-03-29T11:51:46.615 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.599+0000 7f92b6b42640 1 -- 172.21.15.98:0/84184883 learned_addr learned my addr 172.21.15.98:0/84184883 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:51:46.615 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.599+0000 7f92b85a5640 10 monclient(hunting): authenticate will time out at 1013.684903s 2024-03-29T11:51:46.615 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.600+0000 7f92a6ffd640 10 client.?.objecter ms_handle_connect 0x7f92b0142690 2024-03-29T11:51:46.615 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.600+0000 7f92a6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:51:46.615 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.601+0000 7f92a6ffd640 10 client.?.objecter ms_handle_connect 0x7f92b01422b0 2024-03-29T11:51:46.615 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.601+0000 7f92a6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:51:46.615 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.601+0000 7f92a6ffd640 1 -- 172.21.15.98:0/84184883 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 459752648 0 0) 0x7f92b006eac0 con 0x7f92b0142690 2024-03-29T11:51:46.615 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.601+0000 7f92a6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:51:46.615 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.601+0000 7f92a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:46.615 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.601+0000 7f92a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:46.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.601+0000 7f92a6ffd640 10 monclient(hunting): my global_id is 4341 2024-03-29T11:51:46.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.601+0000 7f92a6ffd640 1 -- 172.21.15.98:0/84184883 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9294004eb0 con 0x7f92b0142690 2024-03-29T11:51:46.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.601+0000 7f92a6ffd640 1 -- 172.21.15.98:0/84184883 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1608520214 0 0) 0x7f92b0132320 con 0x7f92b01422b0 2024-03-29T11:51:46.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.601+0000 7f92a6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:51:46.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.601+0000 7f92a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:46.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.601+0000 7f92a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:46.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.601+0000 7f92a6ffd640 10 monclient(hunting): my global_id is 4352 2024-03-29T11:51:46.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.601+0000 7f92a6ffd640 1 -- 172.21.15.98:0/84184883 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92b006eac0 con 0x7f92b01422b0 2024-03-29T11:51:46.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.602+0000 7f92a6ffd640 10 client.?.objecter ms_handle_connect 0x7f92b00b17d0 2024-03-29T11:51:46.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.602+0000 7f92a6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:51:46.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.602+0000 7f92a6ffd640 1 -- 172.21.15.98:0/84184883 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3599243741 0 0) 0x7f92b006eda0 con 0x7f92b00b17d0 2024-03-29T11:51:46.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.602+0000 7f92a6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:51:46.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.602+0000 7f92a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:46.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.602+0000 7f92a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:46.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.602+0000 7f92a6ffd640 10 monclient(hunting): my global_id is 4372 2024-03-29T11:51:46.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.602+0000 7f92a6ffd640 1 -- 172.21.15.98:0/84184883 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92b0132320 con 0x7f92b00b17d0 2024-03-29T11:51:46.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.602+0000 7f92a6ffd640 1 -- 172.21.15.98:0/84184883 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1928372850 0 0) 0x7f9294004eb0 con 0x7f92b0142690 2024-03-29T11:51:46.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.602+0000 7f92a6ffd640 1 -- 172.21.15.98:0/84184883 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92b006eda0 con 0x7f92b0142690 2024-03-29T11:51:46.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.602+0000 7f92a6ffd640 1 -- 172.21.15.98:0/84184883 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f92a00017f0 con 0x7f92b0142690 2024-03-29T11:51:46.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.602+0000 7f92a6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:46.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.602+0000 7f92a6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:51:46.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.602+0000 7f92a6ffd640 10 monclient(hunting): dump: 2024-03-29T11:51:46.617 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:51:46.617 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:46.617 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:46.617 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:46.617 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:46.618 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:46.618 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:51:46.618 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:51:46.618 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:51:46.618 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:46.618 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.602+0000 7f92a6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:51:46.618 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.603+0000 7f92a6ffd640 1 -- 172.21.15.98:0/84184883 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3248594246 0 0) 0x7f92b0132320 con 0x7f92b00b17d0 2024-03-29T11:51:46.618 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.603+0000 7f92a6ffd640 1 -- 172.21.15.98:0/84184883 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9294004eb0 con 0x7f92b00b17d0 2024-03-29T11:51:46.618 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.603+0000 7f92a6ffd640 1 -- 172.21.15.98:0/84184883 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f92ac001b00 con 0x7f92b00b17d0 2024-03-29T11:51:46.618 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.603+0000 7f92a6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:46.618 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.603+0000 7f92a6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:51:46.618 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.603+0000 7f92a6ffd640 10 monclient(hunting): dump: 2024-03-29T11:51:46.618 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:51:46.619 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:46.619 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:46.619 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:46.619 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:46.619 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:46.619 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:51:46.619 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:51:46.619 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:51:46.619 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:46.619 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.603+0000 7f92a6ffd640 1 -- 172.21.15.98:0/84184883 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1575589208 0 0) 0x7f92b006eda0 con 0x7f92b0142690 2024-03-29T11:51:46.619 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.603+0000 7f92a6ffd640 1 -- 172.21.15.98:0/84184883 >> v1:172.21.15.98:6790/0 conn(0x7f92b01422b0 legacy=0x7f92b012ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:46.619 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.603+0000 7f92a6ffd640 1 -- 172.21.15.98:0/84184883 >> v1:172.21.15.121:6789/0 conn(0x7f92b00b17d0 legacy=0x7f92b01dec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:46.619 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.603+0000 7f92a6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:51:46.619 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.603+0000 7f92a6ffd640 1 monclient: found mon.a 2024-03-29T11:51:46.620 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.603+0000 7f92a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:51:46.620 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.603+0000 7f92a6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:51:46.620 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.603+0000 7f92a6ffd640 1 -- 172.21.15.98:0/84184883 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b0132600 con 0x7f92b0142690 2024-03-29T11:51:46.620 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.603+0000 7f92a6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:51:46.620 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.603+0000 7f92a6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:51:46.620 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.603+0000 7f92a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:51:46.620 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.603+0000 7f92b85a5640 5 monclient: authenticate success, global_id 4341 2024-03-29T11:51:46.620 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.603+0000 7f92b85a5640 10 monclient: _renew_subs 2024-03-29T11:51:46.620 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.603+0000 7f92b85a5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:51:46.620 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.603+0000 7f92b85a5640 1 -- 172.21.15.98:0/84184883 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92b0131e00 con 0x7f92b0142690 2024-03-29T11:51:46.620 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.603+0000 7f92b85a5640 10 client.4341.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:51:46.620 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.603+0000 7f92b85a5640 10 monclient: _renew_subs 2024-03-29T11:51:46.620 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.603+0000 7f92b85a5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:51:46.620 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.603+0000 7f92b85a5640 1 -- 172.21.15.98:0/84184883 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92b013bb10 con 0x7f92b0142690 2024-03-29T11:51:46.621 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.603+0000 7f92b85a5640 1 librados: init done 2024-03-29T11:51:46.621 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.604+0000 7f92a6ffd640 1 -- 172.21.15.98:0/84184883 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92a0001950 con 0x7f92b0142690 2024-03-29T11:51:46.621 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.604+0000 7f92a6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:51:46.621 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.604+0000 7f92a6ffd640 1 -- 172.21.15.98:0/84184883 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f92a0004900 con 0x7f92b0142690 2024-03-29T11:51:46.621 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.604+0000 7f92a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:46.621 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.604+0000 7f92a6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:51:46.621 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.604+0000 7f92b4b3e640 4 set_mon_vals no callback set 2024-03-29T11:51:46.621 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.604+0000 7f92a6ffd640 10 monclient: dump: 2024-03-29T11:51:46.621 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:51:46.621 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:46.621 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:46.621 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:46.621 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:46.621 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:46.622 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:51:46.622 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:51:46.622 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:51:46.622 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:46.622 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.606+0000 7f92b85a5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:46.622 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.606+0000 7f92b85a5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:46.622 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.606+0000 7f92b85a5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:51:46.622 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.606+0000 7f92a6ffd640 1 -- 172.21.15.98:0/84184883 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f92a001a8d0 con 0x7f92b0142690 2024-03-29T11:51:46.622 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.606+0000 7f92b85a5640 1 -- 172.21.15.98:0/84184883 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9298005080 con 0x7f92b0142690 2024-03-29T11:51:46.622 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.607+0000 7f92b6341640 10 monclient: get_auth_request con 0x7f929406e060 auth_method 0 2024-03-29T11:51:46.622 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.607+0000 7f92a6ffd640 1 -- 172.21.15.98:0/84184883 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f92a00848e0 con 0x7f92b0142690 2024-03-29T11:51:46.622 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.607+0000 7f92a6ffd640 10 client.4341.objecter ms_dispatch 0x7f92b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:51:46.622 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.607+0000 7f92a6ffd640 3 client.4341.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:51:46.623 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.607+0000 7f92a6ffd640 3 client.4341.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:51:46.623 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.607+0000 7f92a6ffd640 20 client.4341.objecter dump_active .. 0 homeless 2024-03-29T11:51:46.623 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.608+0000 7f92a6ffd640 10 client.4341.objecter ms_handle_connect 0x7f929406e060 2024-03-29T11:51:46.623 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.614+0000 7f92a6ffd640 1 -- 172.21.15.98:0/84184883 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f92a00501c0 con 0x7f92b0142690 2024-03-29T11:51:46.623 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.614+0000 7f92a6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:46.623 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.614+0000 7f92a6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:51:46.978 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.971+0000 7f92b85a5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:46.978 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.971+0000 7f92b85a5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:46.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.971+0000 7f92b85a5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:51:46.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.971+0000 7f92b85a5640 1 -- 172.21.15.98:0/84184883 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9298005330 con 0x7f92b0142690 2024-03-29T11:51:46.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.972+0000 7f92a6ffd640 1 -- 172.21.15.98:0/84184883 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f92a00520f0 con 0x7f92b0142690 2024-03-29T11:51:46.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.972+0000 7f92a6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:46.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.972+0000 7f92a6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:51:46.979 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:51:46.980 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:51:46.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.975+0000 7f92b85a5640 10 librados: watch_flush enter 2024-03-29T11:51:46.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.976+0000 7f92b85a5640 10 librados: watch_flush exit 2024-03-29T11:51:46.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.976+0000 7f92b85a5640 20 client.4341.objecter shutdown clearing up homeless session... 2024-03-29T11:51:46.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.976+0000 7f92b85a5640 10 client.4341.objecter successfully canceled tick 2024-03-29T11:51:46.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.976+0000 7f92b85a5640 1 -- 172.21.15.98:0/84184883 >> v1:172.21.15.98:6816/4065186974 conn(0x7f929406e060 legacy=0x7f9294070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:46.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.976+0000 7f92b85a5640 10 monclient: shutdown 2024-03-29T11:51:46.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.976+0000 7f92b85a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:51:46.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.976+0000 7f92b85a5640 1 -- 172.21.15.98:0/84184883 >> v1:172.21.15.98:6789/0 conn(0x7f92b0142690 legacy=0x7f92b01e6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:46.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.976+0000 7f92b85a5640 1 -- 172.21.15.98:0/84184883 shutdown_connections 2024-03-29T11:51:46.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.976+0000 7f92b85a5640 1 -- 172.21.15.98:0/84184883 >> 172.21.15.98:0/84184883 conn(0x7f92b0066a00 msgr2=0x7f92b0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:51:46.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.976+0000 7f92b85a5640 1 -- 172.21.15.98:0/84184883 shutdown_connections 2024-03-29T11:51:46.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.976+0000 7f92b85a5640 1 -- 172.21.15.98:0/84184883 wait complete. 2024-03-29T11:51:46.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:46.976+0000 7f92b85a5640 1 librados: shutdown 2024-03-29T11:51:46.999 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:51:53.805 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:51:53.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.947+0000 7feb27b77640 10 monclient: get_monmap_and_config 2024-03-29T11:51:53.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.947+0000 7feb27b77640 10 monclient: build_initial_monmap 2024-03-29T11:51:53.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.947+0000 7feb27b77640 10 monclient: monmap: 2024-03-29T11:51:53.956 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:51:53.956 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:53.956 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:51:53.948808+0000 2024-03-29T11:51:53.956 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:51:53.948808+0000 2024-03-29T11:51:53.956 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:51:53.956 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:53.956 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:51:53.956 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:51:53.956 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:51:53.956 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:53.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.949+0000 7feb27b77640 1 Processor -- start 2024-03-29T11:51:53.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.950+0000 7feb27b77640 1 -- start start 2024-03-29T11:51:53.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.950+0000 7feb27b77640 10 monclient: init 2024-03-29T11:51:53.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.951+0000 7feb27b77640 10 monclient: _reopen_session rank -1 2024-03-29T11:51:53.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.951+0000 7feb27b77640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:51:53.958 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.951+0000 7feb27b77640 10 monclient(hunting): picked mon.noname-a con 0x7feb200ae250 addr v1:172.21.15.98:6789/0 2024-03-29T11:51:53.958 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.951+0000 7feb27b77640 10 monclient(hunting): picked mon.noname-c con 0x7feb2013f970 addr v1:172.21.15.121:6789/0 2024-03-29T11:51:53.958 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.951+0000 7feb27b77640 10 monclient(hunting): picked mon.noname-b con 0x7feb20142800 addr v1:172.21.15.98:6790/0 2024-03-29T11:51:53.958 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.951+0000 7feb27b77640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb200afe00 con 0x7feb200ae250 2024-03-29T11:51:53.958 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.951+0000 7feb27b77640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb20142c10 con 0x7feb2013f970 2024-03-29T11:51:53.958 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.951+0000 7feb27b77640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb200afb20 con 0x7feb20142800 2024-03-29T11:51:53.958 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.951+0000 7feb27b77640 10 monclient(hunting): _renew_subs 2024-03-29T11:51:53.958 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.951+0000 7feb27b77640 10 monclient(hunting): authenticate will time out at 1021.036214s 2024-03-29T11:51:53.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.952+0000 7feb26114640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7feb20142800 0x7feb2014da10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:43032/0 (socket says 172.21.15.98:43032) 2024-03-29T11:51:53.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.952+0000 7feb26114640 1 -- 172.21.15.98:0/3365164908 learned_addr learned my addr 172.21.15.98:0/3365164908 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:51:53.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.953+0000 7feb24911640 1 -- 172.21.15.98:0/3365164908 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 630554917 0 0) 0x7feb200afb20 con 0x7feb20142800 2024-03-29T11:51:53.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.953+0000 7feb24911640 10 monclient(hunting): _negotiate 2024-03-29T11:51:53.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.953+0000 7feb24911640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:53.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.953+0000 7feb24911640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:53.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.953+0000 7feb24911640 10 monclient(hunting): my global_id is 4355 2024-03-29T11:51:53.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.953+0000 7feb24911640 1 -- 172.21.15.98:0/3365164908 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb0c003520 con 0x7feb20142800 2024-03-29T11:51:53.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.955+0000 7feb24911640 1 -- 172.21.15.98:0/3365164908 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 91996430 0 0) 0x7feb20142c10 con 0x7feb2013f970 2024-03-29T11:51:53.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.955+0000 7feb24911640 10 monclient(hunting): _negotiate 2024-03-29T11:51:53.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.955+0000 7feb24911640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:53.970 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.955+0000 7feb24911640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:53.970 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.955+0000 7feb24911640 10 monclient(hunting): my global_id is 4375 2024-03-29T11:51:53.970 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.955+0000 7feb24911640 1 -- 172.21.15.98:0/3365164908 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb200afb20 con 0x7feb2013f970 2024-03-29T11:51:53.970 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.955+0000 7feb24911640 1 -- 172.21.15.98:0/3365164908 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 626240278 0 0) 0x7feb200afe00 con 0x7feb200ae250 2024-03-29T11:51:53.970 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.955+0000 7feb24911640 10 monclient(hunting): _negotiate 2024-03-29T11:51:53.970 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.955+0000 7feb24911640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:53.970 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.955+0000 7feb24911640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:53.970 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.955+0000 7feb24911640 10 monclient(hunting): my global_id is 4344 2024-03-29T11:51:53.970 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.955+0000 7feb24911640 1 -- 172.21.15.98:0/3365164908 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb20142c10 con 0x7feb200ae250 2024-03-29T11:51:53.970 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.955+0000 7feb24911640 1 -- 172.21.15.98:0/3365164908 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1780564593 0 0) 0x7feb200afb20 con 0x7feb2013f970 2024-03-29T11:51:53.970 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.955+0000 7feb24911640 1 -- 172.21.15.98:0/3365164908 >> v1:172.21.15.98:6790/0 conn(0x7feb20142800 legacy=0x7feb2014da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:53.970 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.955+0000 7feb24911640 1 -- 172.21.15.98:0/3365164908 >> v1:172.21.15.98:6789/0 conn(0x7feb200ae250 legacy=0x7feb2013f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:53.970 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.955+0000 7feb24911640 10 monclient: _finish_hunting 0 2024-03-29T11:51:53.970 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.955+0000 7feb24911640 1 monclient: found mon.noname-c 2024-03-29T11:51:53.970 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.955+0000 7feb24911640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:51:53.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.955+0000 7feb24911640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:51:53.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.955+0000 7feb24911640 1 -- 172.21.15.98:0/3365164908 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb200aab80 con 0x7feb2013f970 2024-03-29T11:51:53.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.955+0000 7feb24911640 10 monclient: _finish_auth 0 2024-03-29T11:51:53.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.955+0000 7feb24911640 10 monclient: _check_auth_tickets 2024-03-29T11:51:53.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.955+0000 7feb24911640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:51:53.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.955+0000 7feb24911640 1 -- 172.21.15.98:0/3365164908 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7feb1c001670 con 0x7feb2013f970 2024-03-29T11:51:53.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.955+0000 7feb24911640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:53.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.955+0000 7feb24911640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:51:53.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.955+0000 7feb24911640 10 monclient: dump: 2024-03-29T11:51:53.971 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:51:53.971 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:53.971 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:53.971 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:53.971 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:53.971 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:53.971 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:51:53.971 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:51:53.972 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:51:53.972 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:53.972 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.955+0000 7feb27b77640 5 monclient: authenticate success, global_id 4375 2024-03-29T11:51:53.972 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.955+0000 7feb27b77640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:51:53.972 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.955+0000 7feb24911640 1 -- 172.21.15.98:0/3365164908 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb1c001890 con 0x7feb2013f970 2024-03-29T11:51:53.972 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.955+0000 7feb24911640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:51:53.972 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.955+0000 7feb27b77640 10 monclient: get_monmap_and_config success 2024-03-29T11:51:53.973 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.955+0000 7feb27b77640 4 set_mon_vals no callback set 2024-03-29T11:51:53.973 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.956+0000 7feb24911640 1 -- 172.21.15.98:0/3365164908 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7feb1c001c20 con 0x7feb2013f970 2024-03-29T11:51:53.973 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.956+0000 7feb24911640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:53.973 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.956+0000 7feb24911640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:51:53.973 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.956+0000 7feb24911640 10 monclient: dump: 2024-03-29T11:51:53.973 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:51:53.973 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:53.973 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:53.973 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:53.973 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:53.973 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:53.973 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:51:53.973 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:51:53.973 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:51:53.973 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:53.973 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.956+0000 7feb27b77640 10 monclient: shutdown 2024-03-29T11:51:53.973 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.956+0000 7feb27b77640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:51:53.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.956+0000 7feb27b77640 1 -- 172.21.15.98:0/3365164908 >> v1:172.21.15.121:6789/0 conn(0x7feb2013f970 legacy=0x7feb201420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:53.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.958+0000 7feb27b77640 1 -- 172.21.15.98:0/3365164908 shutdown_connections 2024-03-29T11:51:53.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.958+0000 7feb27b77640 1 -- 172.21.15.98:0/3365164908 >> 172.21.15.98:0/3365164908 conn(0x7feb200b35e0 msgr2=0x7feb200b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:51:53.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.958+0000 7feb27b77640 1 -- 172.21.15.98:0/3365164908 shutdown_connections 2024-03-29T11:51:53.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.958+0000 7feb27b77640 1 -- 172.21.15.98:0/3365164908 wait complete. 2024-03-29T11:51:53.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.959+0000 7feb27b77640 10 monclient: build_initial_monmap 2024-03-29T11:51:53.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.959+0000 7feb27b77640 10 monclient: monmap: 2024-03-29T11:51:53.974 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:51:53.974 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:51:53.974 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:51:53.974 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:51:53.974 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:51:53.974 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:53.974 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:51:53.974 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:51:53.974 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:51:53.975 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:53.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.960+0000 7feb27b77640 1 librados: starting msgr at 2024-03-29T11:51:53.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.960+0000 7feb27b77640 1 librados: starting objecter 2024-03-29T11:51:53.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.960+0000 7feb27b77640 1 Processor -- start 2024-03-29T11:51:53.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.960+0000 7feb27b77640 1 -- start start 2024-03-29T11:51:53.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.960+0000 7feb27b77640 1 librados: setting wanted keys 2024-03-29T11:51:53.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.960+0000 7feb27b77640 1 librados: calling monclient init 2024-03-29T11:51:53.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.960+0000 7feb27b77640 10 monclient: init 2024-03-29T11:51:53.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.961+0000 7feb27b77640 10 monclient: _reopen_session rank -1 2024-03-29T11:51:53.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.961+0000 7feb27b77640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:51:53.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.961+0000 7feb27b77640 10 monclient(hunting): picked mon.noname-c con 0x7feb200ae250 addr v1:172.21.15.98:6790/0 2024-03-29T11:51:53.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.962+0000 7feb27b77640 10 monclient(hunting): picked mon.noname-b con 0x7feb2013f970 addr v1:172.21.15.121:6789/0 2024-03-29T11:51:53.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.962+0000 7feb27b77640 10 monclient(hunting): picked mon.noname-a con 0x7feb20142800 addr v1:172.21.15.98:6789/0 2024-03-29T11:51:53.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.962+0000 7feb27b77640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb2013c1d0 con 0x7feb20142800 2024-03-29T11:51:53.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.962+0000 7feb27b77640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb20143f20 con 0x7feb2013f970 2024-03-29T11:51:53.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.962+0000 7feb27b77640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb200acb80 con 0x7feb200ae250 2024-03-29T11:51:53.976 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.962+0000 7feb27b77640 10 monclient(hunting): _renew_subs 2024-03-29T11:51:53.976 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.962+0000 7feb27b77640 10 monclient(hunting): authenticate will time out at 1021.047333s 2024-03-29T11:51:53.976 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.962+0000 7feb25913640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7feb200ae250 0x7feb201d71a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:43344/0 (socket says 172.21.15.98:43344) 2024-03-29T11:51:53.976 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.962+0000 7feb25913640 1 -- 172.21.15.98:0/470212129 learned_addr learned my addr 172.21.15.98:0/470212129 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:51:53.976 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.962+0000 7feb167fc640 10 client.?.objecter ms_handle_connect 0x7feb200ae250 2024-03-29T11:51:53.976 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.962+0000 7feb167fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:51:53.976 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.962+0000 7feb167fc640 1 -- 172.21.15.98:0/470212129 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 366700286 0 0) 0x7feb200acb80 con 0x7feb200ae250 2024-03-29T11:51:53.976 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.962+0000 7feb167fc640 10 monclient(hunting): _negotiate 2024-03-29T11:51:53.976 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.962+0000 7feb167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:51:53.976 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.962+0000 7feb167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:51:53.976 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.962+0000 7feb167fc640 10 monclient(hunting): my global_id is 4358 2024-03-29T11:51:53.976 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.962+0000 7feb167fc640 1 -- 172.21.15.98:0/470212129 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb04004eb0 con 0x7feb200ae250 2024-03-29T11:51:53.978 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.963+0000 7feb167fc640 1 -- 172.21.15.98:0/470212129 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3483459414 0 0) 0x7feb04004eb0 con 0x7feb200ae250 2024-03-29T11:51:53.978 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.963+0000 7feb167fc640 1 -- 172.21.15.98:0/470212129 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb200acb80 con 0x7feb200ae250 2024-03-29T11:51:53.978 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.963+0000 7feb167fc640 1 -- 172.21.15.98:0/470212129 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7feb10001a30 con 0x7feb200ae250 2024-03-29T11:51:53.978 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.963+0000 7feb167fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:53.978 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.963+0000 7feb167fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:51:53.978 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.963+0000 7feb167fc640 10 monclient(hunting): dump: 2024-03-29T11:51:53.978 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:51:53.978 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:53.979 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:53.979 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:53.979 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:53.979 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:53.979 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:51:53.979 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:51:53.979 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:51:53.979 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:53.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.963+0000 7feb167fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:51:53.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.967+0000 7feb167fc640 1 -- 172.21.15.98:0/470212129 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4157705736 0 0) 0x7feb200acb80 con 0x7feb200ae250 2024-03-29T11:51:53.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.967+0000 7feb167fc640 1 -- 172.21.15.98:0/470212129 >> v1:172.21.15.121:6789/0 conn(0x7feb2013f970 legacy=0x7feb201459f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:51:53.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.967+0000 7feb167fc640 1 -- 172.21.15.98:0/470212129 >> v1:172.21.15.98:6789/0 conn(0x7feb20142800 legacy=0x7feb201e2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:53.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.967+0000 7feb167fc640 10 monclient: _finish_hunting 0 2024-03-29T11:51:53.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.967+0000 7feb167fc640 1 monclient: found mon.c 2024-03-29T11:51:53.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.967+0000 7feb167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:51:53.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.967+0000 7feb167fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:51:53.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.967+0000 7feb167fc640 1 -- 172.21.15.98:0/470212129 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb200acd10 con 0x7feb200ae250 2024-03-29T11:51:53.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.967+0000 7feb167fc640 10 monclient: _finish_auth 0 2024-03-29T11:51:53.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.967+0000 7feb167fc640 10 monclient: _check_auth_tickets 2024-03-29T11:51:53.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.967+0000 7feb167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:51:53.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.967+0000 7feb27b77640 5 monclient: authenticate success, global_id 4358 2024-03-29T11:51:53.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.967+0000 7feb27b77640 10 monclient: _renew_subs 2024-03-29T11:51:53.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.967+0000 7feb27b77640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:51:53.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.967+0000 7feb27b77640 1 -- 172.21.15.98:0/470212129 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb2013b440 con 0x7feb200ae250 2024-03-29T11:51:53.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.967+0000 7feb27b77640 10 client.4358.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:51:53.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.967+0000 7feb27b77640 10 monclient: _renew_subs 2024-03-29T11:51:53.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.967+0000 7feb27b77640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:51:53.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.967+0000 7feb27b77640 1 -- 172.21.15.98:0/470212129 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feb201d78b0 con 0x7feb200ae250 2024-03-29T11:51:53.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.967+0000 7feb27b77640 1 librados: init done 2024-03-29T11:51:53.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.968+0000 7feb167fc640 1 -- 172.21.15.98:0/470212129 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb10001b90 con 0x7feb200ae250 2024-03-29T11:51:53.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.968+0000 7feb167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:51:53.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.968+0000 7feb17fff640 4 set_mon_vals no callback set 2024-03-29T11:51:53.981 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.969+0000 7feb167fc640 1 -- 172.21.15.98:0/470212129 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7feb10003a60 con 0x7feb200ae250 2024-03-29T11:51:53.981 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.969+0000 7feb167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:51:53.981 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.969+0000 7feb167fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:51:53.981 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.969+0000 7feb167fc640 10 monclient: dump: 2024-03-29T11:51:53.981 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:51:53.981 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:51:53.981 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:53.981 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:51:53.981 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:51:53.981 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:51:53.981 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:51:53.981 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:51:53.981 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:51:53.981 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:51:53.981 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.969+0000 7feb167fc640 1 -- 172.21.15.98:0/470212129 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7feb10003c80 con 0x7feb200ae250 2024-03-29T11:51:53.981 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.970+0000 7feb27b77640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:53.982 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.970+0000 7feb27b77640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:53.982 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.970+0000 7feb27b77640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:51:53.982 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.970+0000 7feb27b77640 1 -- 172.21.15.98:0/470212129 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb08005080 con 0x7feb200ae250 2024-03-29T11:51:53.982 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.970+0000 7feb25112640 10 monclient: get_auth_request con 0x7feb0406db90 auth_method 0 2024-03-29T11:51:53.982 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.970+0000 7feb167fc640 1 -- 172.21.15.98:0/470212129 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7feb10084bb0 con 0x7feb200ae250 2024-03-29T11:51:53.982 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.970+0000 7feb167fc640 10 client.4358.objecter ms_dispatch 0x7feb20000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:51:53.982 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.970+0000 7feb167fc640 3 client.4358.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:51:53.982 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.970+0000 7feb167fc640 3 client.4358.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:51:53.982 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.971+0000 7feb167fc640 20 client.4358.objecter dump_active .. 0 homeless 2024-03-29T11:51:53.982 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.971+0000 7feb167fc640 10 client.4358.objecter ms_handle_connect 0x7feb0406db90 2024-03-29T11:51:53.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.980+0000 7feb167fc640 1 -- 172.21.15.98:0/470212129 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feb10050490 con 0x7feb200ae250 2024-03-29T11:51:53.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.980+0000 7feb167fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:51:53.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:53.980+0000 7feb167fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:51:54.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:54.303+0000 7feb27b77640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:54.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:54.303+0000 7feb27b77640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:54.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:54.303+0000 7feb27b77640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:51:54.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:54.303+0000 7feb27b77640 1 -- 172.21.15.98:0/470212129 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb08005330 con 0x7feb200ae250 2024-03-29T11:51:54.312 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:54.304+0000 7feb167fc640 1 -- 172.21.15.98:0/470212129 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7feb100523c0 con 0x7feb200ae250 2024-03-29T11:51:54.312 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:54.304+0000 7feb167fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:51:54.312 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:54.304+0000 7feb167fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:51:54.312 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:51:54.312 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:51:54.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:54.308+0000 7feb27b77640 10 librados: watch_flush enter 2024-03-29T11:51:54.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:54.308+0000 7feb27b77640 10 librados: watch_flush exit 2024-03-29T11:51:54.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:54.308+0000 7feb27b77640 20 client.4358.objecter shutdown clearing up homeless session... 2024-03-29T11:51:54.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:54.308+0000 7feb27b77640 10 client.4358.objecter successfully canceled tick 2024-03-29T11:51:54.318 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:54.309+0000 7feb27b77640 1 -- 172.21.15.98:0/470212129 >> v1:172.21.15.98:6816/4065186974 conn(0x7feb0406db90 legacy=0x7feb04070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:54.318 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:54.309+0000 7feb27b77640 10 monclient: shutdown 2024-03-29T11:51:54.318 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:54.309+0000 7feb27b77640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:51:54.318 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:54.309+0000 7feb27b77640 1 -- 172.21.15.98:0/470212129 >> v1:172.21.15.98:6790/0 conn(0x7feb200ae250 legacy=0x7feb201d71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:51:54.318 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:54.309+0000 7feb27b77640 1 -- 172.21.15.98:0/470212129 shutdown_connections 2024-03-29T11:51:54.318 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:54.309+0000 7feb27b77640 1 -- 172.21.15.98:0/470212129 >> 172.21.15.98:0/470212129 conn(0x7feb200b35e0 msgr2=0x7feb200b1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:51:54.318 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:54.311+0000 7feb27b77640 1 -- 172.21.15.98:0/470212129 shutdown_connections 2024-03-29T11:51:54.319 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:54.311+0000 7feb27b77640 1 -- 172.21.15.98:0/470212129 wait complete. 2024-03-29T11:51:54.319 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:51:54.311+0000 7feb27b77640 1 librados: shutdown 2024-03-29T11:51:54.336 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:52:01.143 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:52:01.291 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.284+0000 7fb19e605640 10 monclient: get_monmap_and_config 2024-03-29T11:52:01.291 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.284+0000 7fb19e605640 10 monclient: build_initial_monmap 2024-03-29T11:52:01.292 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.284+0000 7fb19e605640 10 monclient: monmap: 2024-03-29T11:52:01.292 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:52:01.292 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:01.292 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:52:01.285510+0000 2024-03-29T11:52:01.292 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:52:01.285510+0000 2024-03-29T11:52:01.292 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:52:01.292 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:01.292 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:52:01.292 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:52:01.292 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:52:01.292 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:01.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.285+0000 7fb19e605640 1 Processor -- start 2024-03-29T11:52:01.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.285+0000 7fb19e605640 1 -- start start 2024-03-29T11:52:01.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.285+0000 7fb19e605640 10 monclient: init 2024-03-29T11:52:01.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.286+0000 7fb19e605640 10 monclient: _reopen_session rank -1 2024-03-29T11:52:01.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.286+0000 7fb19e605640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:52:01.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.286+0000 7fb19e605640 10 monclient(hunting): picked mon.noname-c con 0x7fb1980a9930 addr v1:172.21.15.121:6789/0 2024-03-29T11:52:01.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.286+0000 7fb19e605640 10 monclient(hunting): picked mon.noname-b con 0x7fb198149dc0 addr v1:172.21.15.98:6790/0 2024-03-29T11:52:01.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.286+0000 7fb19e605640 10 monclient(hunting): picked mon.noname-a con 0x7fb19814ff40 addr v1:172.21.15.98:6789/0 2024-03-29T11:52:01.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.286+0000 7fb19e605640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1980aae60 con 0x7fb19814ff40 2024-03-29T11:52:01.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.286+0000 7fb19e605640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1980b4310 con 0x7fb1980a9930 2024-03-29T11:52:01.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.286+0000 7fb19e605640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1980afdd0 con 0x7fb198149dc0 2024-03-29T11:52:01.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.286+0000 7fb19e605640 10 monclient(hunting): _renew_subs 2024-03-29T11:52:01.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.286+0000 7fb19e605640 10 monclient(hunting): authenticate will time out at 1028.371668s 2024-03-29T11:52:01.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.289+0000 7fb19cba2640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fb19814ff40 0x7fb198152330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38177/0 (socket says 172.21.15.98:38177) 2024-03-29T11:52:01.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.289+0000 7fb19cba2640 1 -- 172.21.15.98:0/1583478533 learned_addr learned my addr 172.21.15.98:0/1583478533 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:52:01.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.290+0000 7fb196ffd640 1 -- 172.21.15.98:0/1583478533 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 969022706 0 0) 0x7fb1980aae60 con 0x7fb19814ff40 2024-03-29T11:52:01.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.290+0000 7fb196ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:52:01.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.290+0000 7fb196ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:01.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.290+0000 7fb196ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:01.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.290+0000 7fb196ffd640 10 monclient(hunting): my global_id is 4347 2024-03-29T11:52:01.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.290+0000 7fb196ffd640 1 -- 172.21.15.98:0/1583478533 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb180003520 con 0x7fb19814ff40 2024-03-29T11:52:01.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.290+0000 7fb196ffd640 1 -- 172.21.15.98:0/1583478533 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2586419456 0 0) 0x7fb1980afdd0 con 0x7fb198149dc0 2024-03-29T11:52:01.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.290+0000 7fb196ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:52:01.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.290+0000 7fb196ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:01.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.290+0000 7fb196ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:01.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.290+0000 7fb196ffd640 10 monclient(hunting): my global_id is 4361 2024-03-29T11:52:01.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.290+0000 7fb196ffd640 1 -- 172.21.15.98:0/1583478533 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1980aae60 con 0x7fb198149dc0 2024-03-29T11:52:01.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.290+0000 7fb196ffd640 1 -- 172.21.15.98:0/1583478533 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1448603409 0 0) 0x7fb1980aae60 con 0x7fb198149dc0 2024-03-29T11:52:01.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.290+0000 7fb196ffd640 1 -- 172.21.15.98:0/1583478533 >> v1:172.21.15.121:6789/0 conn(0x7fb1980a9930 legacy=0x7fb198143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:01.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.290+0000 7fb196ffd640 1 -- 172.21.15.98:0/1583478533 >> v1:172.21.15.98:6789/0 conn(0x7fb19814ff40 legacy=0x7fb198152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:01.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.290+0000 7fb196ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:52:01.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.290+0000 7fb196ffd640 1 monclient: found mon.noname-b 2024-03-29T11:52:01.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.290+0000 7fb196ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:52:01.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.290+0000 7fb196ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:52:01.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.290+0000 7fb196ffd640 1 -- 172.21.15.98:0/1583478533 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb19814a950 con 0x7fb198149dc0 2024-03-29T11:52:01.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.290+0000 7fb196ffd640 10 monclient: _finish_auth 0 2024-03-29T11:52:01.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.290+0000 7fb196ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:52:01.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.290+0000 7fb196ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:52:01.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.290+0000 7fb196ffd640 1 -- 172.21.15.98:0/1583478533 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fb1880015a0 con 0x7fb198149dc0 2024-03-29T11:52:01.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.290+0000 7fb196ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:01.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.290+0000 7fb196ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:52:01.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.290+0000 7fb196ffd640 10 monclient: dump: 2024-03-29T11:52:01.305 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:52:01.305 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:01.305 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:01.305 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:01.305 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:01.305 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:01.305 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:52:01.305 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:52:01.305 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:52:01.306 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:01.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.290+0000 7fb19e605640 5 monclient: authenticate success, global_id 4361 2024-03-29T11:52:01.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.290+0000 7fb19e605640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:52:01.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.291+0000 7fb196ffd640 1 -- 172.21.15.98:0/1583478533 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb188001700 con 0x7fb198149dc0 2024-03-29T11:52:01.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.291+0000 7fb196ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:52:01.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.291+0000 7fb19e605640 10 monclient: get_monmap_and_config success 2024-03-29T11:52:01.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.291+0000 7fb19e605640 4 set_mon_vals no callback set 2024-03-29T11:52:01.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.292+0000 7fb19e605640 10 monclient: shutdown 2024-03-29T11:52:01.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.292+0000 7fb19e605640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:52:01.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.292+0000 7fb19e605640 1 -- 172.21.15.98:0/1583478533 >> v1:172.21.15.98:6790/0 conn(0x7fb198149dc0 legacy=0x7fb19814a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:01.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.292+0000 7fb19e605640 1 -- 172.21.15.98:0/1583478533 shutdown_connections 2024-03-29T11:52:01.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.292+0000 7fb19e605640 1 -- 172.21.15.98:0/1583478533 >> 172.21.15.98:0/1583478533 conn(0x7fb1980b35e0 msgr2=0x7fb1980b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:52:01.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.295+0000 7fb19e605640 1 -- 172.21.15.98:0/1583478533 shutdown_connections 2024-03-29T11:52:01.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.295+0000 7fb19e605640 1 -- 172.21.15.98:0/1583478533 wait complete. 2024-03-29T11:52:01.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.295+0000 7fb19e605640 10 monclient: build_initial_monmap 2024-03-29T11:52:01.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.295+0000 7fb19e605640 10 monclient: monmap: 2024-03-29T11:52:01.308 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:52:01.308 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:52:01.308 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:52:01.309 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:52:01.309 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:52:01.309 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:01.309 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:52:01.309 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:52:01.309 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:52:01.309 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:01.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.296+0000 7fb19e605640 1 librados: starting msgr at 2024-03-29T11:52:01.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.296+0000 7fb19e605640 1 librados: starting objecter 2024-03-29T11:52:01.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.296+0000 7fb19e605640 1 Processor -- start 2024-03-29T11:52:01.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.296+0000 7fb19e605640 1 -- start start 2024-03-29T11:52:01.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.296+0000 7fb19e605640 1 librados: setting wanted keys 2024-03-29T11:52:01.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.296+0000 7fb19e605640 1 librados: calling monclient init 2024-03-29T11:52:01.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.296+0000 7fb19e605640 10 monclient: init 2024-03-29T11:52:01.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.297+0000 7fb19e605640 10 monclient: _reopen_session rank -1 2024-03-29T11:52:01.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.297+0000 7fb19e605640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:52:01.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.297+0000 7fb19e605640 10 monclient(hunting): picked mon.noname-a con 0x7fb1980a9930 addr v1:172.21.15.98:6789/0 2024-03-29T11:52:01.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.297+0000 7fb19e605640 10 monclient(hunting): picked mon.noname-c con 0x7fb198149dc0 addr v1:172.21.15.98:6790/0 2024-03-29T11:52:01.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.297+0000 7fb19e605640 10 monclient(hunting): picked mon.noname-b con 0x7fb19814ff40 addr v1:172.21.15.121:6789/0 2024-03-29T11:52:01.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.297+0000 7fb19e605640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb19814aae0 con 0x7fb1980a9930 2024-03-29T11:52:01.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.297+0000 7fb19e605640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb19814bc00 con 0x7fb19814ff40 2024-03-29T11:52:01.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.297+0000 7fb19e605640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1980710e0 con 0x7fb198149dc0 2024-03-29T11:52:01.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.297+0000 7fb19e605640 10 monclient(hunting): _renew_subs 2024-03-29T11:52:01.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.297+0000 7fb19e605640 10 monclient(hunting): authenticate will time out at 1028.382555s 2024-03-29T11:52:01.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.298+0000 7fb1977fe640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fb198149dc0 0x7fb1980718c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:59906/0 (socket says 172.21.15.98:59906) 2024-03-29T11:52:01.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.298+0000 7fb1977fe640 1 -- 172.21.15.98:0/1203300236 learned_addr learned my addr 172.21.15.98:0/1203300236 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:52:01.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.298+0000 7fb194ff9640 10 client.?.objecter ms_handle_connect 0x7fb198149dc0 2024-03-29T11:52:01.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.298+0000 7fb194ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:52:01.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.298+0000 7fb194ff9640 1 -- 172.21.15.98:0/1203300236 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 603367148 0 0) 0x7fb1980710e0 con 0x7fb198149dc0 2024-03-29T11:52:01.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.298+0000 7fb194ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:52:01.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.298+0000 7fb194ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:01.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.298+0000 7fb194ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:01.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.298+0000 7fb194ff9640 10 monclient(hunting): my global_id is 4364 2024-03-29T11:52:01.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.298+0000 7fb194ff9640 1 -- 172.21.15.98:0/1203300236 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb184004eb0 con 0x7fb198149dc0 2024-03-29T11:52:01.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.299+0000 7fb194ff9640 1 -- 172.21.15.98:0/1203300236 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3088942825 0 0) 0x7fb184004eb0 con 0x7fb198149dc0 2024-03-29T11:52:01.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.299+0000 7fb194ff9640 1 -- 172.21.15.98:0/1203300236 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1980710e0 con 0x7fb198149dc0 2024-03-29T11:52:01.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.299+0000 7fb194ff9640 1 -- 172.21.15.98:0/1203300236 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fb188001580 con 0x7fb198149dc0 2024-03-29T11:52:01.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.299+0000 7fb194ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:01.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.299+0000 7fb194ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:52:01.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.299+0000 7fb194ff9640 10 monclient(hunting): dump: 2024-03-29T11:52:01.311 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:52:01.311 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:01.311 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:01.311 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:01.312 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:01.312 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:01.312 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:52:01.312 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:52:01.312 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:52:01.312 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:01.312 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.299+0000 7fb194ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:52:01.312 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.299+0000 7fb194ff9640 1 -- 172.21.15.98:0/1203300236 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3085971295 0 0) 0x7fb1980710e0 con 0x7fb198149dc0 2024-03-29T11:52:01.312 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.299+0000 7fb194ff9640 1 -- 172.21.15.98:0/1203300236 >> v1:172.21.15.121:6789/0 conn(0x7fb19814ff40 legacy=0x7fb198071fd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:52:01.312 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.300+0000 7fb194ff9640 1 -- 172.21.15.98:0/1203300236 >> v1:172.21.15.98:6789/0 conn(0x7fb1980a9930 legacy=0x7fb1981d5030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:52:01.315 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.303+0000 7fb194ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:52:01.315 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.303+0000 7fb194ff9640 1 monclient: found mon.c 2024-03-29T11:52:01.315 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.303+0000 7fb194ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:52:01.315 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.303+0000 7fb194ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:52:01.315 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.303+0000 7fb194ff9640 1 -- 172.21.15.98:0/1203300236 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb198071600 con 0x7fb198149dc0 2024-03-29T11:52:01.315 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.303+0000 7fb194ff9640 10 monclient: _finish_auth 0 2024-03-29T11:52:01.315 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.303+0000 7fb194ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:52:01.315 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.303+0000 7fb194ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:52:01.315 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.303+0000 7fb194ff9640 1 -- 172.21.15.98:0/1203300236 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb188003a60 con 0x7fb198149dc0 2024-03-29T11:52:01.315 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.303+0000 7fb194ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:52:01.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.303+0000 7fb194ff9640 1 -- 172.21.15.98:0/1203300236 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fb188003eb0 con 0x7fb198149dc0 2024-03-29T11:52:01.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.303+0000 7fb194ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:01.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.303+0000 7fb194ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:52:01.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.303+0000 7fb194ff9640 10 monclient: dump: 2024-03-29T11:52:01.316 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:52:01.316 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:01.316 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:01.316 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:01.316 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:01.316 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:01.316 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:52:01.316 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:52:01.316 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:52:01.316 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:01.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.304+0000 7fb19e605640 5 monclient: authenticate success, global_id 4364 2024-03-29T11:52:01.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.304+0000 7fb19e605640 10 monclient: _renew_subs 2024-03-29T11:52:01.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.304+0000 7fb19e605640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:52:01.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.304+0000 7fb19e605640 1 -- 172.21.15.98:0/1203300236 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1981d5740 con 0x7fb198149dc0 2024-03-29T11:52:01.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.304+0000 7fb19e605640 10 client.4364.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:52:01.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.304+0000 7fb19e605640 10 monclient: _renew_subs 2024-03-29T11:52:01.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.304+0000 7fb19e605640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:52:01.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.304+0000 7fb19e605640 1 -- 172.21.15.98:0/1203300236 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1981d5a90 con 0x7fb198149dc0 2024-03-29T11:52:01.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.304+0000 7fb19e605640 1 librados: init done 2024-03-29T11:52:01.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.304+0000 7fb1967fc640 4 set_mon_vals no callback set 2024-03-29T11:52:01.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.306+0000 7fb194ff9640 1 -- 172.21.15.98:0/1203300236 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7fb188004010 con 0x7fb198149dc0 2024-03-29T11:52:01.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.306+0000 7fb197fff640 10 monclient: get_auth_request con 0x7fb18406dc90 auth_method 0 2024-03-29T11:52:01.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.307+0000 7fb194ff9640 1 -- 172.21.15.98:0/1203300236 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7fb188084c40 con 0x7fb198149dc0 2024-03-29T11:52:01.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.307+0000 7fb194ff9640 10 client.4364.objecter ms_dispatch 0x7fb198000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:52:01.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.307+0000 7fb194ff9640 3 client.4364.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:52:01.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.307+0000 7fb194ff9640 3 client.4364.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:52:01.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.307+0000 7fb194ff9640 20 client.4364.objecter dump_active .. 0 homeless 2024-03-29T11:52:01.318 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.307+0000 7fb194ff9640 10 client.4364.objecter ms_handle_connect 0x7fb18406dc90 2024-03-29T11:52:01.318 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.307+0000 7fb19e605640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:01.318 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.307+0000 7fb19e605640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:01.318 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.307+0000 7fb19e605640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:52:01.318 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.307+0000 7fb19e605640 1 -- 172.21.15.98:0/1203300236 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb198075f10 con 0x7fb198149dc0 2024-03-29T11:52:01.324 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.317+0000 7fb194ff9640 1 -- 172.21.15.98:0/1203300236 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb188050520 con 0x7fb198149dc0 2024-03-29T11:52:01.324 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.317+0000 7fb194ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:01.324 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.317+0000 7fb194ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:52:01.646 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.639+0000 7fb19e605640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:01.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.639+0000 7fb19e605640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:01.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.640+0000 7fb19e605640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:52:01.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.640+0000 7fb19e605640 1 -- 172.21.15.98:0/1203300236 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb198071270 con 0x7fb198149dc0 2024-03-29T11:52:01.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.640+0000 7fb194ff9640 1 -- 172.21.15.98:0/1203300236 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7fb188052450 con 0x7fb198149dc0 2024-03-29T11:52:01.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.641+0000 7fb194ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:01.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.641+0000 7fb194ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:52:01.648 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:52:01.648 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:52:01.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.645+0000 7fb19e605640 10 librados: watch_flush enter 2024-03-29T11:52:01.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.645+0000 7fb19e605640 10 librados: watch_flush exit 2024-03-29T11:52:01.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.645+0000 7fb19e605640 20 client.4364.objecter shutdown clearing up homeless session... 2024-03-29T11:52:01.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.645+0000 7fb19e605640 10 client.4364.objecter successfully canceled tick 2024-03-29T11:52:01.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.646+0000 7fb19e605640 1 -- 172.21.15.98:0/1203300236 >> v1:172.21.15.98:6816/4065186974 conn(0x7fb18406dc90 legacy=0x7fb184070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:01.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.646+0000 7fb19e605640 10 monclient: shutdown 2024-03-29T11:52:01.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.646+0000 7fb19e605640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:52:01.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.646+0000 7fb19e605640 1 -- 172.21.15.98:0/1203300236 >> v1:172.21.15.98:6790/0 conn(0x7fb198149dc0 legacy=0x7fb1980718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:01.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.646+0000 7fb19e605640 1 -- 172.21.15.98:0/1203300236 shutdown_connections 2024-03-29T11:52:01.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.646+0000 7fb19e605640 1 -- 172.21.15.98:0/1203300236 >> 172.21.15.98:0/1203300236 conn(0x7fb1980b35e0 msgr2=0x7fb1980b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:52:01.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.647+0000 7fb19e605640 1 -- 172.21.15.98:0/1203300236 shutdown_connections 2024-03-29T11:52:01.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.647+0000 7fb19e605640 1 -- 172.21.15.98:0/1203300236 wait complete. 2024-03-29T11:52:01.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:01.647+0000 7fb19e605640 1 librados: shutdown 2024-03-29T11:52:01.671 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:52:08.478 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:52:08.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.604+0000 7fc831c2e640 10 monclient: get_monmap_and_config 2024-03-29T11:52:08.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.604+0000 7fc831c2e640 10 monclient: build_initial_monmap 2024-03-29T11:52:08.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.604+0000 7fc831c2e640 10 monclient: monmap: 2024-03-29T11:52:08.611 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:52:08.611 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:08.611 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:52:08.604946+0000 2024-03-29T11:52:08.611 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:52:08.604946+0000 2024-03-29T11:52:08.611 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:52:08.611 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:08.611 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:52:08.611 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:52:08.611 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:52:08.612 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:08.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.605+0000 7fc831c2e640 1 Processor -- start 2024-03-29T11:52:08.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.605+0000 7fc831c2e640 1 -- start start 2024-03-29T11:52:08.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.605+0000 7fc831c2e640 10 monclient: init 2024-03-29T11:52:08.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.605+0000 7fc831c2e640 10 monclient: _reopen_session rank -1 2024-03-29T11:52:08.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.605+0000 7fc831c2e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:52:08.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.605+0000 7fc831c2e640 10 monclient(hunting): picked mon.noname-a con 0x7fc82c1457f0 addr v1:172.21.15.98:6789/0 2024-03-29T11:52:08.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.605+0000 7fc831c2e640 10 monclient(hunting): picked mon.noname-b con 0x7fc82c13f650 addr v1:172.21.15.98:6790/0 2024-03-29T11:52:08.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.605+0000 7fc831c2e640 10 monclient(hunting): picked mon.noname-c con 0x7fc82c0a9d70 addr v1:172.21.15.121:6789/0 2024-03-29T11:52:08.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.605+0000 7fc831c2e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc82c0aaef0 con 0x7fc82c1457f0 2024-03-29T11:52:08.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.605+0000 7fc831c2e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc82c145bd0 con 0x7fc82c0a9d70 2024-03-29T11:52:08.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.605+0000 7fc831c2e640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc82c145eb0 con 0x7fc82c13f650 2024-03-29T11:52:08.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.605+0000 7fc831c2e640 10 monclient(hunting): _renew_subs 2024-03-29T11:52:08.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.605+0000 7fc831c2e640 10 monclient(hunting): authenticate will time out at 1035.690962s 2024-03-29T11:52:08.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.606+0000 7fc82affd640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fc82c13f650 0x7fc82c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:50802/0 (socket says 172.21.15.98:50802) 2024-03-29T11:52:08.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.606+0000 7fc82affd640 1 -- 172.21.15.98:0/2061323468 learned_addr learned my addr 172.21.15.98:0/2061323468 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:52:08.615 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.607+0000 7fc82a7fc640 1 -- 172.21.15.98:0/2061323468 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2976788546 0 0) 0x7fc82c145bd0 con 0x7fc82c0a9d70 2024-03-29T11:52:08.615 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.607+0000 7fc82a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:52:08.615 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.607+0000 7fc82a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:08.615 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.607+0000 7fc82a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:08.615 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.607+0000 7fc82a7fc640 10 monclient(hunting): my global_id is 4381 2024-03-29T11:52:08.615 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.607+0000 7fc82a7fc640 1 -- 172.21.15.98:0/2061323468 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc814003520 con 0x7fc82c0a9d70 2024-03-29T11:52:08.615 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.607+0000 7fc82a7fc640 1 -- 172.21.15.98:0/2061323468 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 900488424 0 0) 0x7fc82c145eb0 con 0x7fc82c13f650 2024-03-29T11:52:08.615 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.607+0000 7fc82a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:52:08.615 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.608+0000 7fc82a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:08.615 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.608+0000 7fc82a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:08.615 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.608+0000 7fc82a7fc640 10 monclient(hunting): my global_id is 4367 2024-03-29T11:52:08.615 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.608+0000 7fc82a7fc640 1 -- 172.21.15.98:0/2061323468 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc82c145bd0 con 0x7fc82c13f650 2024-03-29T11:52:08.615 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.608+0000 7fc82a7fc640 1 -- 172.21.15.98:0/2061323468 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1822879436 0 0) 0x7fc82c145bd0 con 0x7fc82c13f650 2024-03-29T11:52:08.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.608+0000 7fc82a7fc640 1 -- 172.21.15.98:0/2061323468 >> v1:172.21.15.121:6789/0 conn(0x7fc82c0a9d70 legacy=0x7fc82c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:08.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.608+0000 7fc82a7fc640 1 -- 172.21.15.98:0/2061323468 >> v1:172.21.15.98:6789/0 conn(0x7fc82c1457f0 legacy=0x7fc82c13ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:52:08.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.609+0000 7fc82a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:52:08.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.609+0000 7fc82a7fc640 1 monclient: found mon.noname-b 2024-03-29T11:52:08.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.609+0000 7fc82a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:52:08.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.609+0000 7fc82a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:52:08.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.609+0000 7fc82a7fc640 1 -- 172.21.15.98:0/2061323468 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc82c146340 con 0x7fc82c13f650 2024-03-29T11:52:08.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.609+0000 7fc82a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:52:08.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.609+0000 7fc82a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:52:08.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.609+0000 7fc82a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:52:08.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.609+0000 7fc82a7fc640 1 -- 172.21.15.98:0/2061323468 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fc820001670 con 0x7fc82c13f650 2024-03-29T11:52:08.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.609+0000 7fc82a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:08.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.609+0000 7fc82a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:52:08.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.609+0000 7fc82a7fc640 10 monclient: dump: 2024-03-29T11:52:08.617 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:52:08.617 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:08.617 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:08.617 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:08.617 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:08.617 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:08.617 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:52:08.618 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:52:08.618 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:52:08.618 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:08.618 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.609+0000 7fc831c2e640 5 monclient: authenticate success, global_id 4367 2024-03-29T11:52:08.618 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.609+0000 7fc831c2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:52:08.620 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.610+0000 7fc82a7fc640 1 -- 172.21.15.98:0/2061323468 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8200017d0 con 0x7fc82c13f650 2024-03-29T11:52:08.620 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.610+0000 7fc82a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:52:08.620 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.610+0000 7fc82a7fc640 1 -- 172.21.15.98:0/2061323468 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fc820001be0 con 0x7fc82c13f650 2024-03-29T11:52:08.620 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.610+0000 7fc831c2e640 10 monclient: get_monmap_and_config success 2024-03-29T11:52:08.620 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.610+0000 7fc831c2e640 4 set_mon_vals no callback set 2024-03-29T11:52:08.620 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.611+0000 7fc82a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:08.620 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.611+0000 7fc82a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:52:08.621 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.611+0000 7fc82a7fc640 10 monclient: dump: 2024-03-29T11:52:08.621 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:52:08.621 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:08.621 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:08.621 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:08.621 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:08.621 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:08.621 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:52:08.621 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:52:08.621 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:52:08.621 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:08.621 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.611+0000 7fc831c2e640 10 monclient: shutdown 2024-03-29T11:52:08.621 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.611+0000 7fc831c2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:52:08.621 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.611+0000 7fc831c2e640 1 -- 172.21.15.98:0/2061323468 >> v1:172.21.15.98:6790/0 conn(0x7fc82c13f650 legacy=0x7fc82c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:08.621 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.611+0000 7fc831c2e640 1 -- 172.21.15.98:0/2061323468 shutdown_connections 2024-03-29T11:52:08.622 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.611+0000 7fc831c2e640 1 -- 172.21.15.98:0/2061323468 >> 172.21.15.98:0/2061323468 conn(0x7fc82c066a00 msgr2=0x7fc82c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:52:08.622 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.611+0000 7fc831c2e640 1 -- 172.21.15.98:0/2061323468 shutdown_connections 2024-03-29T11:52:08.622 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.611+0000 7fc831c2e640 1 -- 172.21.15.98:0/2061323468 wait complete. 2024-03-29T11:52:08.622 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.612+0000 7fc831c2e640 10 monclient: build_initial_monmap 2024-03-29T11:52:08.622 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.612+0000 7fc831c2e640 10 monclient: monmap: 2024-03-29T11:52:08.622 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:52:08.622 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:52:08.622 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:52:08.622 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:52:08.622 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:52:08.622 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:08.623 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:52:08.623 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:52:08.623 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:52:08.623 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:08.623 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.613+0000 7fc831c2e640 1 librados: starting msgr at 2024-03-29T11:52:08.623 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.613+0000 7fc831c2e640 1 librados: starting objecter 2024-03-29T11:52:08.623 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.613+0000 7fc831c2e640 1 Processor -- start 2024-03-29T11:52:08.623 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.613+0000 7fc831c2e640 1 -- start start 2024-03-29T11:52:08.623 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.613+0000 7fc831c2e640 1 librados: setting wanted keys 2024-03-29T11:52:08.623 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.613+0000 7fc831c2e640 1 librados: calling monclient init 2024-03-29T11:52:08.623 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.613+0000 7fc831c2e640 10 monclient: init 2024-03-29T11:52:08.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.614+0000 7fc831c2e640 10 monclient: _reopen_session rank -1 2024-03-29T11:52:08.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.614+0000 7fc831c2e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:52:08.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.614+0000 7fc831c2e640 10 monclient(hunting): picked mon.noname-c con 0x7fc82c0a9d70 addr v1:172.21.15.98:6790/0 2024-03-29T11:52:08.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.614+0000 7fc82b7fe640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fc82c0a9d70 0x7fc82c136e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:51506/0 (socket says 172.21.15.98:51506) 2024-03-29T11:52:08.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.614+0000 7fc82b7fe640 1 -- 172.21.15.98:0/1974626900 learned_addr learned my addr 172.21.15.98:0/1974626900 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:52:08.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.615+0000 7fc813fff640 10 client.?.objecter ms_handle_connect 0x7fc82c0a9d70 2024-03-29T11:52:08.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.615+0000 7fc813fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:52:08.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.615+0000 7fc831c2e640 10 monclient(hunting): picked mon.noname-b con 0x7fc82c13f650 addr v1:172.21.15.121:6789/0 2024-03-29T11:52:08.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.615+0000 7fc831c2e640 10 monclient(hunting): picked mon.noname-a con 0x7fc82c1457f0 addr v1:172.21.15.98:6789/0 2024-03-29T11:52:08.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.615+0000 7fc813fff640 10 client.?.objecter ms_handle_connect 0x7fc82c13f650 2024-03-29T11:52:08.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.615+0000 7fc813fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:52:08.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.615+0000 7fc831c2e640 1 -- 172.21.15.98:0/1974626900 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc82c1428a0 con 0x7fc82c1457f0 2024-03-29T11:52:08.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.615+0000 7fc831c2e640 1 -- 172.21.15.98:0/1974626900 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc82c141ed0 con 0x7fc82c13f650 2024-03-29T11:52:08.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.615+0000 7fc831c2e640 1 -- 172.21.15.98:0/1974626900 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc82c141a30 con 0x7fc82c0a9d70 2024-03-29T11:52:08.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.615+0000 7fc831c2e640 10 monclient(hunting): _renew_subs 2024-03-29T11:52:08.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.615+0000 7fc813fff640 1 -- 172.21.15.98:0/1974626900 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3316821900 0 0) 0x7fc82c141ed0 con 0x7fc82c13f650 2024-03-29T11:52:08.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.616+0000 7fc831c2e640 10 monclient(hunting): authenticate will time out at 1035.701063s 2024-03-29T11:52:08.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.616+0000 7fc813fff640 10 monclient(hunting): _negotiate 2024-03-29T11:52:08.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.616+0000 7fc813fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:08.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.616+0000 7fc813fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:08.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.616+0000 7fc813fff640 10 monclient(hunting): my global_id is 4384 2024-03-29T11:52:08.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.616+0000 7fc813fff640 1 -- 172.21.15.98:0/1974626900 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc818004eb0 con 0x7fc82c13f650 2024-03-29T11:52:08.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.616+0000 7fc813fff640 1 -- 172.21.15.98:0/1974626900 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 517034943 0 0) 0x7fc82c141a30 con 0x7fc82c0a9d70 2024-03-29T11:52:08.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.616+0000 7fc813fff640 10 monclient(hunting): _negotiate 2024-03-29T11:52:08.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.616+0000 7fc813fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:08.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.616+0000 7fc813fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:08.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.616+0000 7fc813fff640 10 monclient(hunting): my global_id is 4370 2024-03-29T11:52:08.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.616+0000 7fc813fff640 1 -- 172.21.15.98:0/1974626900 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc82c141ed0 con 0x7fc82c0a9d70 2024-03-29T11:52:08.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.616+0000 7fc813fff640 1 -- 172.21.15.98:0/1974626900 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 91476072 0 0) 0x7fc818004eb0 con 0x7fc82c13f650 2024-03-29T11:52:08.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.616+0000 7fc813fff640 1 -- 172.21.15.98:0/1974626900 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc82c141a30 con 0x7fc82c13f650 2024-03-29T11:52:08.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.616+0000 7fc813fff640 1 -- 172.21.15.98:0/1974626900 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fc820001ba0 con 0x7fc82c13f650 2024-03-29T11:52:08.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.616+0000 7fc813fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:08.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.616+0000 7fc813fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:52:08.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.616+0000 7fc813fff640 10 monclient(hunting): dump: 2024-03-29T11:52:08.632 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:52:08.632 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:08.632 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:08.632 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:08.632 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:08.632 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:08.632 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:52:08.632 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:52:08.632 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:52:08.632 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:08.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.616+0000 7fc813fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:52:08.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.616+0000 7fc813fff640 1 -- 172.21.15.98:0/1974626900 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 234425409 0 0) 0x7fc82c141ed0 con 0x7fc82c0a9d70 2024-03-29T11:52:08.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.616+0000 7fc813fff640 1 -- 172.21.15.98:0/1974626900 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc818004eb0 con 0x7fc82c0a9d70 2024-03-29T11:52:08.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.616+0000 7fc813fff640 1 -- 172.21.15.98:0/1974626900 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fc81c003fa0 con 0x7fc82c0a9d70 2024-03-29T11:52:08.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.616+0000 7fc813fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:08.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.616+0000 7fc813fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:52:08.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.616+0000 7fc813fff640 10 monclient(hunting): dump: 2024-03-29T11:52:08.633 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:52:08.633 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:08.633 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:08.633 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:08.633 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:08.633 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:08.633 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:52:08.633 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:52:08.633 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:52:08.633 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:08.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.617+0000 7fc813fff640 1 -- 172.21.15.98:0/1974626900 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3680634082 0 0) 0x7fc818004eb0 con 0x7fc82c0a9d70 2024-03-29T11:52:08.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.617+0000 7fc813fff640 1 -- 172.21.15.98:0/1974626900 >> v1:172.21.15.121:6789/0 conn(0x7fc82c13f650 legacy=0x7fc82c1da440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:08.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.617+0000 7fc813fff640 1 -- 172.21.15.98:0/1974626900 >> v1:172.21.15.98:6789/0 conn(0x7fc82c1457f0 legacy=0x7fc82c1dab50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:52:08.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.618+0000 7fc813fff640 10 monclient: _finish_hunting 0 2024-03-29T11:52:08.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.618+0000 7fc813fff640 1 monclient: found mon.c 2024-03-29T11:52:08.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.618+0000 7fc813fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:52:08.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.618+0000 7fc813fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:52:08.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.618+0000 7fc813fff640 1 -- 172.21.15.98:0/1974626900 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc82c141bc0 con 0x7fc82c0a9d70 2024-03-29T11:52:08.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.618+0000 7fc813fff640 10 monclient: _finish_auth 0 2024-03-29T11:52:08.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.618+0000 7fc813fff640 10 monclient: _check_auth_tickets 2024-03-29T11:52:08.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.618+0000 7fc813fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:52:08.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.618+0000 7fc831c2e640 5 monclient: authenticate success, global_id 4370 2024-03-29T11:52:08.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.618+0000 7fc831c2e640 10 monclient: _renew_subs 2024-03-29T11:52:08.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.618+0000 7fc831c2e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:52:08.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.618+0000 7fc831c2e640 1 -- 172.21.15.98:0/1974626900 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc82c137540 con 0x7fc82c0a9d70 2024-03-29T11:52:08.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.618+0000 7fc831c2e640 10 client.4370.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:52:08.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.618+0000 7fc831c2e640 10 monclient: _renew_subs 2024-03-29T11:52:08.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.618+0000 7fc831c2e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:52:08.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.618+0000 7fc831c2e640 1 -- 172.21.15.98:0/1974626900 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc82c1447c0 con 0x7fc82c0a9d70 2024-03-29T11:52:08.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.618+0000 7fc831c2e640 1 librados: init done 2024-03-29T11:52:08.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.619+0000 7fc831c2e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:08.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.619+0000 7fc831c2e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:08.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.619+0000 7fc831c2e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:52:08.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.619+0000 7fc831c2e640 1 -- 172.21.15.98:0/1974626900 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc82c142a50 con 0x7fc82c0a9d70 2024-03-29T11:52:08.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.620+0000 7fc813fff640 1 -- 172.21.15.98:0/1974626900 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc81c0044b0 con 0x7fc82c0a9d70 2024-03-29T11:52:08.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.620+0000 7fc813fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:52:08.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.620+0000 7fc813fff640 1 -- 172.21.15.98:0/1974626900 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fc81c001070 con 0x7fc82c0a9d70 2024-03-29T11:52:08.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.620+0000 7fc813fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:08.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.620+0000 7fc813fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:52:08.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.620+0000 7fc813fff640 10 monclient: dump: 2024-03-29T11:52:08.635 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:52:08.636 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:08.636 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:08.636 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:08.636 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:08.636 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:08.636 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:52:08.636 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:52:08.636 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:52:08.636 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:08.636 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.620+0000 7fc829ffb640 4 set_mon_vals no callback set 2024-03-29T11:52:08.636 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.621+0000 7fc813fff640 1 -- 172.21.15.98:0/1974626900 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7fc81c001290 con 0x7fc82c0a9d70 2024-03-29T11:52:08.636 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.621+0000 7fc813fff640 1 -- 172.21.15.98:0/1974626900 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7fc81c0849c0 con 0x7fc82c0a9d70 2024-03-29T11:52:08.636 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.621+0000 7fc813fff640 10 client.4370.objecter ms_dispatch 0x7fc82c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:52:08.636 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.621+0000 7fc813fff640 3 client.4370.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:52:08.636 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.621+0000 7fc813fff640 3 client.4370.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:52:08.636 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.621+0000 7fc813fff640 20 client.4370.objecter dump_active .. 0 homeless 2024-03-29T11:52:08.637 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.622+0000 7fc82affd640 10 monclient: get_auth_request con 0x7fc81806de40 auth_method 0 2024-03-29T11:52:08.637 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.622+0000 7fc813fff640 10 client.4370.objecter ms_handle_connect 0x7fc81806de40 2024-03-29T11:52:08.637 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.629+0000 7fc813fff640 1 -- 172.21.15.98:0/1974626900 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc82c142a50 con 0x7fc82c0a9d70 2024-03-29T11:52:08.637 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.629+0000 7fc813fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:08.637 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.629+0000 7fc813fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:52:08.945 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:52:08.945 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:52:08.946 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.935+0000 7fc831c2e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:08.946 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.935+0000 7fc831c2e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:08.946 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.935+0000 7fc831c2e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:52:08.946 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.935+0000 7fc831c2e640 1 -- 172.21.15.98:0/1974626900 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc82c142a50 con 0x7fc82c0a9d70 2024-03-29T11:52:08.946 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.936+0000 7fc813fff640 1 -- 172.21.15.98:0/1974626900 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7fc82c142a50 con 0x7fc82c0a9d70 2024-03-29T11:52:08.946 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.936+0000 7fc813fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:08.946 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.936+0000 7fc813fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:52:08.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.939+0000 7fc831c2e640 10 librados: watch_flush enter 2024-03-29T11:52:08.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.939+0000 7fc831c2e640 10 librados: watch_flush exit 2024-03-29T11:52:08.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.939+0000 7fc831c2e640 20 client.4370.objecter shutdown clearing up homeless session... 2024-03-29T11:52:08.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.939+0000 7fc831c2e640 10 client.4370.objecter successfully canceled tick 2024-03-29T11:52:08.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.940+0000 7fc831c2e640 1 -- 172.21.15.98:0/1974626900 >> v1:172.21.15.98:6816/4065186974 conn(0x7fc81806de40 legacy=0x7fc818070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:08.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.940+0000 7fc831c2e640 10 monclient: shutdown 2024-03-29T11:52:08.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.940+0000 7fc831c2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:52:08.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.940+0000 7fc831c2e640 1 -- 172.21.15.98:0/1974626900 >> v1:172.21.15.98:6790/0 conn(0x7fc82c0a9d70 legacy=0x7fc82c136e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:08.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.940+0000 7fc831c2e640 1 -- 172.21.15.98:0/1974626900 shutdown_connections 2024-03-29T11:52:08.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.940+0000 7fc831c2e640 1 -- 172.21.15.98:0/1974626900 >> 172.21.15.98:0/1974626900 conn(0x7fc82c066a00 msgr2=0x7fc82c0b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:52:08.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.940+0000 7fc831c2e640 1 -- 172.21.15.98:0/1974626900 shutdown_connections 2024-03-29T11:52:08.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.940+0000 7fc831c2e640 1 -- 172.21.15.98:0/1974626900 wait complete. 2024-03-29T11:52:08.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:08.940+0000 7fc831c2e640 1 librados: shutdown 2024-03-29T11:52:08.964 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:52:15.771 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:52:15.917 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.910+0000 7fca19786640 10 monclient: get_monmap_and_config 2024-03-29T11:52:15.917 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.910+0000 7fca19786640 10 monclient: build_initial_monmap 2024-03-29T11:52:15.917 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.910+0000 7fca19786640 10 monclient: monmap: 2024-03-29T11:52:15.917 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:52:15.917 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:15.917 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:52:15.911046+0000 2024-03-29T11:52:15.917 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:52:15.911046+0000 2024-03-29T11:52:15.917 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:52:15.917 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:15.917 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:52:15.917 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:52:15.918 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:52:15.918 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:15.918 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.911+0000 7fca19786640 1 Processor -- start 2024-03-29T11:52:15.918 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.911+0000 7fca19786640 1 -- start start 2024-03-29T11:52:15.918 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.911+0000 7fca19786640 10 monclient: init 2024-03-29T11:52:15.919 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.912+0000 7fca19786640 10 monclient: _reopen_session rank -1 2024-03-29T11:52:15.919 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.912+0000 7fca19786640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:52:15.921 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.912+0000 7fca19786640 10 monclient(hunting): picked mon.noname-a con 0x7fca0c0389e0 addr v1:172.21.15.98:6789/0 2024-03-29T11:52:15.921 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.914+0000 7fca19786640 10 monclient(hunting): picked mon.noname-c con 0x7fca0c038dc0 addr v1:172.21.15.121:6789/0 2024-03-29T11:52:15.921 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.914+0000 7fca19786640 10 monclient(hunting): picked mon.noname-b con 0x7fca0c030660 addr v1:172.21.15.98:6790/0 2024-03-29T11:52:15.921 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.914+0000 7fca19786640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0c0c54c0 con 0x7fca0c0389e0 2024-03-29T11:52:15.921 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.914+0000 7fca19786640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0c036430 con 0x7fca0c038dc0 2024-03-29T11:52:15.922 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.914+0000 7fca19786640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0c039840 con 0x7fca0c030660 2024-03-29T11:52:15.922 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.914+0000 7fca19786640 10 monclient(hunting): _renew_subs 2024-03-29T11:52:15.922 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.914+0000 7fca19786640 10 monclient(hunting): authenticate will time out at 1042.999785s 2024-03-29T11:52:15.922 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.914+0000 7fca18f85640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fca0c030660 0x7fca0c0d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:33718/0 (socket says 172.21.15.98:33718) 2024-03-29T11:52:15.922 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.914+0000 7fca18f85640 1 -- 172.21.15.98:0/2731744175 learned_addr learned my addr 172.21.15.98:0/2731744175 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:52:15.923 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.915+0000 7fca12ffd640 1 -- 172.21.15.98:0/2731744175 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1495516061 0 0) 0x7fca0c039840 con 0x7fca0c030660 2024-03-29T11:52:15.923 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.915+0000 7fca12ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:52:15.923 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.915+0000 7fca12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:15.923 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.915+0000 7fca12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:15.923 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.915+0000 7fca12ffd640 10 monclient(hunting): my global_id is 4373 2024-03-29T11:52:15.923 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.915+0000 7fca12ffd640 1 -- 172.21.15.98:0/2731744175 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca04003520 con 0x7fca0c030660 2024-03-29T11:52:15.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.918+0000 7fca12ffd640 1 -- 172.21.15.98:0/2731744175 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1102534439 0 0) 0x7fca0c0c54c0 con 0x7fca0c0389e0 2024-03-29T11:52:15.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.918+0000 7fca12ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:52:15.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.918+0000 7fca12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:15.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.918+0000 7fca12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:15.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.918+0000 7fca12ffd640 10 monclient(hunting): my global_id is 4350 2024-03-29T11:52:15.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.918+0000 7fca12ffd640 1 -- 172.21.15.98:0/2731744175 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca0c039840 con 0x7fca0c0389e0 2024-03-29T11:52:15.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.918+0000 7fca12ffd640 1 -- 172.21.15.98:0/2731744175 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 654781700 0 0) 0x7fca0c039840 con 0x7fca0c0389e0 2024-03-29T11:52:15.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.918+0000 7fca12ffd640 1 -- 172.21.15.98:0/2731744175 >> v1:172.21.15.98:6790/0 conn(0x7fca0c030660 legacy=0x7fca0c0d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:15.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.918+0000 7fca12ffd640 1 -- 172.21.15.98:0/2731744175 >> v1:172.21.15.121:6789/0 conn(0x7fca0c038dc0 legacy=0x7fca0c0cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:15.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.918+0000 7fca12ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:52:15.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.918+0000 7fca12ffd640 1 monclient: found mon.noname-a 2024-03-29T11:52:15.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.918+0000 7fca12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:52:15.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.918+0000 7fca12ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T11:52:15.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.918+0000 7fca12ffd640 1 -- 172.21.15.98:0/2731744175 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca0c035690 con 0x7fca0c0389e0 2024-03-29T11:52:15.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.918+0000 7fca12ffd640 10 monclient: _finish_auth 0 2024-03-29T11:52:15.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.918+0000 7fca12ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:52:15.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.918+0000 7fca12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:52:15.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.918+0000 7fca12ffd640 1 -- 172.21.15.98:0/2731744175 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fca00001690 con 0x7fca0c0389e0 2024-03-29T11:52:15.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.918+0000 7fca12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:15.931 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.918+0000 7fca12ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:52:15.931 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.918+0000 7fca12ffd640 10 monclient: dump: 2024-03-29T11:52:15.931 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:52:15.931 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:15.931 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:15.931 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:15.931 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:15.931 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:15.931 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:52:15.931 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:52:15.931 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:52:15.931 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:15.931 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.920+0000 7fca19786640 5 monclient: authenticate success, global_id 4350 2024-03-29T11:52:15.931 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.920+0000 7fca19786640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:52:15.931 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.920+0000 7fca12ffd640 1 -- 172.21.15.98:0/2731744175 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca00001860 con 0x7fca0c0389e0 2024-03-29T11:52:15.931 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.920+0000 7fca12ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:52:15.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.920+0000 7fca12ffd640 1 -- 172.21.15.98:0/2731744175 <== mon.0 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fca00001c70 con 0x7fca0c0389e0 2024-03-29T11:52:15.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.920+0000 7fca12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:15.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.920+0000 7fca12ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:52:15.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.920+0000 7fca12ffd640 10 monclient: dump: 2024-03-29T11:52:15.932 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:52:15.932 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:15.932 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:15.932 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:15.932 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:15.932 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:15.932 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:52:15.932 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:52:15.932 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:52:15.932 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:15.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.920+0000 7fca19786640 10 monclient: get_monmap_and_config success 2024-03-29T11:52:15.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.920+0000 7fca19786640 4 set_mon_vals no callback set 2024-03-29T11:52:15.933 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.921+0000 7fca19786640 10 monclient: shutdown 2024-03-29T11:52:15.933 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.921+0000 7fca19786640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:52:15.933 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.921+0000 7fca19786640 1 -- 172.21.15.98:0/2731744175 >> v1:172.21.15.98:6789/0 conn(0x7fca0c0389e0 legacy=0x7fca0c033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:15.933 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.922+0000 7fca19786640 1 -- 172.21.15.98:0/2731744175 shutdown_connections 2024-03-29T11:52:15.933 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.922+0000 7fca19786640 1 -- 172.21.15.98:0/2731744175 >> 172.21.15.98:0/2731744175 conn(0x7fca0c005c00 msgr2=0x7fca0c006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:52:15.933 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.922+0000 7fca19786640 1 -- 172.21.15.98:0/2731744175 shutdown_connections 2024-03-29T11:52:15.933 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.922+0000 7fca19786640 1 -- 172.21.15.98:0/2731744175 wait complete. 2024-03-29T11:52:15.934 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.924+0000 7fca19786640 10 monclient: build_initial_monmap 2024-03-29T11:52:15.934 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.924+0000 7fca19786640 10 monclient: monmap: 2024-03-29T11:52:15.934 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:52:15.935 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:52:15.935 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:52:15.935 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:52:15.935 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:52:15.935 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:15.935 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:52:15.935 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:52:15.935 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:52:15.935 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:15.935 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.924+0000 7fca19786640 1 librados: starting msgr at 2024-03-29T11:52:15.935 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.924+0000 7fca19786640 1 librados: starting objecter 2024-03-29T11:52:15.935 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.924+0000 7fca19786640 1 Processor -- start 2024-03-29T11:52:15.935 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.925+0000 7fca19786640 1 -- start start 2024-03-29T11:52:15.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.925+0000 7fca19786640 1 librados: setting wanted keys 2024-03-29T11:52:15.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.925+0000 7fca19786640 1 librados: calling monclient init 2024-03-29T11:52:15.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.925+0000 7fca19786640 10 monclient: init 2024-03-29T11:52:15.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.926+0000 7fca19786640 10 monclient: _reopen_session rank -1 2024-03-29T11:52:15.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.926+0000 7fca19786640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:52:15.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.927+0000 7fca19786640 10 monclient(hunting): picked mon.noname-a con 0x7fca0c030660 addr v1:172.21.15.98:6789/0 2024-03-29T11:52:15.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.927+0000 7fca19786640 10 monclient(hunting): picked mon.noname-c con 0x7fca0c0389e0 addr v1:172.21.15.98:6790/0 2024-03-29T11:52:15.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.927+0000 7fca19786640 10 monclient(hunting): picked mon.noname-b con 0x7fca0c15b7e0 addr v1:172.21.15.121:6789/0 2024-03-29T11:52:15.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.927+0000 7fca19786640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0c0bcdd0 con 0x7fca0c030660 2024-03-29T11:52:15.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.927+0000 7fca19786640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0c15bcc0 con 0x7fca0c15b7e0 2024-03-29T11:52:15.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.927+0000 7fca19786640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0c15c1e0 con 0x7fca0c0389e0 2024-03-29T11:52:15.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.927+0000 7fca19786640 10 monclient(hunting): _renew_subs 2024-03-29T11:52:15.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.927+0000 7fca19786640 10 monclient(hunting): authenticate will time out at 1043.012624s 2024-03-29T11:52:15.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.927+0000 7fca137fe640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fca0c0389e0 0x7fca0c032770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:34928/0 (socket says 172.21.15.98:34928) 2024-03-29T11:52:15.937 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.928+0000 7fca137fe640 1 -- 172.21.15.98:0/1990300137 learned_addr learned my addr 172.21.15.98:0/1990300137 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:52:15.943 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.928+0000 7fca10ff9640 10 client.?.objecter ms_handle_connect 0x7fca0c0389e0 2024-03-29T11:52:15.943 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.928+0000 7fca10ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:52:15.943 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.929+0000 7fca10ff9640 1 -- 172.21.15.98:0/1990300137 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2969225512 0 0) 0x7fca0c15c1e0 con 0x7fca0c0389e0 2024-03-29T11:52:15.943 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.929+0000 7fca10ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:52:15.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.929+0000 7fca10ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:15.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.929+0000 7fca10ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:15.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.929+0000 7fca10ff9640 10 monclient(hunting): my global_id is 4376 2024-03-29T11:52:15.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.929+0000 7fca10ff9640 1 -- 172.21.15.98:0/1990300137 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9f8004eb0 con 0x7fca0c0389e0 2024-03-29T11:52:15.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.929+0000 7fca10ff9640 10 client.?.objecter ms_handle_connect 0x7fca0c030660 2024-03-29T11:52:15.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.929+0000 7fca10ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:52:15.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.929+0000 7fca10ff9640 1 -- 172.21.15.98:0/1990300137 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 291389927 0 0) 0x7fca0c0bcdd0 con 0x7fca0c030660 2024-03-29T11:52:15.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.929+0000 7fca10ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:52:15.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.930+0000 7fca10ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:15.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.930+0000 7fca10ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:15.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.930+0000 7fca10ff9640 10 monclient(hunting): my global_id is 4353 2024-03-29T11:52:15.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.930+0000 7fca10ff9640 1 -- 172.21.15.98:0/1990300137 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca0c15c1e0 con 0x7fca0c030660 2024-03-29T11:52:15.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.930+0000 7fca10ff9640 1 -- 172.21.15.98:0/1990300137 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3608985783 0 0) 0x7fca0c15c1e0 con 0x7fca0c030660 2024-03-29T11:52:15.945 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.930+0000 7fca10ff9640 1 -- 172.21.15.98:0/1990300137 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca0c0bcdd0 con 0x7fca0c030660 2024-03-29T11:52:15.945 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.930+0000 7fca10ff9640 1 -- 172.21.15.98:0/1990300137 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fca00001b70 con 0x7fca0c030660 2024-03-29T11:52:15.945 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.930+0000 7fca10ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:15.945 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.930+0000 7fca10ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:52:15.945 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.930+0000 7fca10ff9640 10 monclient(hunting): dump: 2024-03-29T11:52:15.945 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:52:15.945 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:15.945 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:15.945 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:15.945 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:15.945 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:15.945 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:52:15.945 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:52:15.945 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:52:15.945 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:15.945 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.930+0000 7fca10ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:52:15.946 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.930+0000 7fca10ff9640 10 client.?.objecter ms_handle_connect 0x7fca0c15b7e0 2024-03-29T11:52:15.946 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.930+0000 7fca10ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:52:15.946 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.930+0000 7fca10ff9640 1 -- 172.21.15.98:0/1990300137 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1590037620 0 0) 0x7fca0c0bcdd0 con 0x7fca0c030660 2024-03-29T11:52:15.946 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.930+0000 7fca10ff9640 1 -- 172.21.15.98:0/1990300137 >> v1:172.21.15.98:6790/0 conn(0x7fca0c0389e0 legacy=0x7fca0c032770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:15.946 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.930+0000 7fca10ff9640 1 -- 172.21.15.98:0/1990300137 >> v1:172.21.15.121:6789/0 conn(0x7fca0c15b7e0 legacy=0x7fca0c162b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:15.946 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.930+0000 7fca10ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:52:15.946 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.930+0000 7fca10ff9640 1 monclient: found mon.a 2024-03-29T11:52:15.946 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.931+0000 7fca10ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:52:15.946 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.931+0000 7fca10ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:52:15.946 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.931+0000 7fca10ff9640 1 -- 172.21.15.98:0/1990300137 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca0c15c370 con 0x7fca0c030660 2024-03-29T11:52:15.946 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.931+0000 7fca10ff9640 10 monclient: _finish_auth 0 2024-03-29T11:52:15.946 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.931+0000 7fca10ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:52:15.946 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.931+0000 7fca10ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:52:15.946 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.931+0000 7fca19786640 5 monclient: authenticate success, global_id 4353 2024-03-29T11:52:15.946 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.931+0000 7fca19786640 10 monclient: _renew_subs 2024-03-29T11:52:15.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.931+0000 7fca19786640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:52:15.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.931+0000 7fca19786640 1 -- 172.21.15.98:0/1990300137 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca0c15c5c0 con 0x7fca0c030660 2024-03-29T11:52:15.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.931+0000 7fca19786640 10 client.4353.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:52:15.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.931+0000 7fca19786640 10 monclient: _renew_subs 2024-03-29T11:52:15.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.931+0000 7fca19786640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:52:15.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.931+0000 7fca19786640 1 -- 172.21.15.98:0/1990300137 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca0c164430 con 0x7fca0c030660 2024-03-29T11:52:15.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.931+0000 7fca19786640 1 librados: init done 2024-03-29T11:52:15.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.931+0000 7fca10ff9640 1 -- 172.21.15.98:0/1990300137 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca00001d50 con 0x7fca0c030660 2024-03-29T11:52:15.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.932+0000 7fca10ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:52:15.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.932+0000 7fca10ff9640 1 -- 172.21.15.98:0/1990300137 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fca00003ae0 con 0x7fca0c030660 2024-03-29T11:52:15.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.932+0000 7fca10ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:15.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.932+0000 7fca10ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:52:15.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.932+0000 7fca10ff9640 10 monclient: dump: 2024-03-29T11:52:15.947 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:52:15.947 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:15.947 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:15.948 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:15.948 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:15.948 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:15.948 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:52:15.948 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:52:15.948 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:52:15.948 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:15.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.932+0000 7fca127fc640 4 set_mon_vals no callback set 2024-03-29T11:52:15.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.933+0000 7fca19786640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:15.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.933+0000 7fca19786640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:15.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.934+0000 7fca19786640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:52:15.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.934+0000 7fca19786640 1 -- 172.21.15.98:0/1990300137 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9f4005080 con 0x7fca0c030660 2024-03-29T11:52:15.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.934+0000 7fca10ff9640 1 -- 172.21.15.98:0/1990300137 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7fca00019ab0 con 0x7fca0c030660 2024-03-29T11:52:15.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.934+0000 7fca10ff9640 1 -- 172.21.15.98:0/1990300137 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7fca00083950 con 0x7fca0c030660 2024-03-29T11:52:15.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.934+0000 7fca10ff9640 10 client.4353.objecter ms_dispatch 0x7fca14000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:52:15.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.934+0000 7fca10ff9640 3 client.4353.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:52:15.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.934+0000 7fca10ff9640 3 client.4353.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:52:15.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.935+0000 7fca137fe640 10 monclient: get_auth_request con 0x7fc9f806dc00 auth_method 0 2024-03-29T11:52:15.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.935+0000 7fca10ff9640 20 client.4353.objecter dump_active .. 0 homeless 2024-03-29T11:52:15.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.935+0000 7fca10ff9640 10 client.4353.objecter ms_handle_connect 0x7fc9f806dc00 2024-03-29T11:52:15.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.942+0000 7fca10ff9640 1 -- 172.21.15.98:0/1990300137 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fca0004f230 con 0x7fca0c030660 2024-03-29T11:52:15.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.942+0000 7fca10ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:15.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:15.942+0000 7fca10ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:52:16.283 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:16.276+0000 7fca19786640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:16.283 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:16.276+0000 7fca19786640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:16.284 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:16.276+0000 7fca19786640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:52:16.284 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:16.276+0000 7fca19786640 1 -- 172.21.15.98:0/1990300137 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9f4005330 con 0x7fca0c030660 2024-03-29T11:52:16.285 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:16.277+0000 7fca10ff9640 1 -- 172.21.15.98:0/1990300137 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7fca00051160 con 0x7fca0c030660 2024-03-29T11:52:16.285 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:16.277+0000 7fca10ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:16.285 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:16.277+0000 7fca10ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:52:16.285 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:52:16.285 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:52:16.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:16.281+0000 7fca19786640 10 librados: watch_flush enter 2024-03-29T11:52:16.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:16.281+0000 7fca19786640 10 librados: watch_flush exit 2024-03-29T11:52:16.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:16.281+0000 7fca19786640 20 client.4353.objecter shutdown clearing up homeless session... 2024-03-29T11:52:16.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:16.281+0000 7fca19786640 10 client.4353.objecter successfully canceled tick 2024-03-29T11:52:16.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:16.281+0000 7fca19786640 1 -- 172.21.15.98:0/1990300137 >> v1:172.21.15.98:6816/4065186974 conn(0x7fc9f806dc00 legacy=0x7fc9f80700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:16.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:16.281+0000 7fca19786640 10 monclient: shutdown 2024-03-29T11:52:16.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:16.281+0000 7fca19786640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:52:16.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:16.281+0000 7fca19786640 1 -- 172.21.15.98:0/1990300137 >> v1:172.21.15.98:6789/0 conn(0x7fca0c030660 legacy=0x7fca0c032060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:16.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:16.281+0000 7fca19786640 1 -- 172.21.15.98:0/1990300137 shutdown_connections 2024-03-29T11:52:16.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:16.281+0000 7fca19786640 1 -- 172.21.15.98:0/1990300137 >> 172.21.15.98:0/1990300137 conn(0x7fca0c005c00 msgr2=0x7fca0c0c2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:52:16.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:16.281+0000 7fca19786640 1 -- 172.21.15.98:0/1990300137 shutdown_connections 2024-03-29T11:52:16.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:16.281+0000 7fca19786640 1 -- 172.21.15.98:0/1990300137 wait complete. 2024-03-29T11:52:16.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:16.281+0000 7fca19786640 1 librados: shutdown 2024-03-29T11:52:16.306 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:52:23.113 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:52:23.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.246+0000 7f748ffff640 10 monclient: get_monmap_and_config 2024-03-29T11:52:23.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.246+0000 7f748ffff640 10 monclient: build_initial_monmap 2024-03-29T11:52:23.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.246+0000 7f748ffff640 10 monclient: monmap: 2024-03-29T11:52:23.253 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:52:23.253 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:23.253 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:52:23.246961+0000 2024-03-29T11:52:23.254 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:52:23.246961+0000 2024-03-29T11:52:23.254 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:52:23.254 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:23.254 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:52:23.254 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:52:23.254 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:52:23.254 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:23.254 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.247+0000 7f748ffff640 1 Processor -- start 2024-03-29T11:52:23.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.247+0000 7f748ffff640 1 -- start start 2024-03-29T11:52:23.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.247+0000 7f748ffff640 10 monclient: init 2024-03-29T11:52:23.256 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.248+0000 7f748ffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:52:23.256 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.248+0000 7f748ffff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:52:23.256 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.248+0000 7f748effd640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f74900b4920 0x7f749013f290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:55236/0 (socket says 172.21.15.98:55236) 2024-03-29T11:52:23.256 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.248+0000 7f748effd640 1 -- 172.21.15.98:0/2084943187 learned_addr learned my addr 172.21.15.98:0/2084943187 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:52:23.263 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.248+0000 7f748ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f74900b4920 addr v1:172.21.15.98:6790/0 2024-03-29T11:52:23.263 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.248+0000 7f748ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f74900b4200 addr v1:172.21.15.121:6789/0 2024-03-29T11:52:23.263 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.248+0000 7f748ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f749013f9a0 addr v1:172.21.15.98:6789/0 2024-03-29T11:52:23.264 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.248+0000 7f748ffff640 1 -- 172.21.15.98:0/2084943187 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74900b0930 con 0x7f749013f9a0 2024-03-29T11:52:23.264 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.248+0000 7f748ffff640 1 -- 172.21.15.98:0/2084943187 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74900af460 con 0x7f74900b4200 2024-03-29T11:52:23.264 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.250+0000 7f748ffff640 1 -- 172.21.15.98:0/2084943187 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74900aeda0 con 0x7f74900b4920 2024-03-29T11:52:23.264 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.250+0000 7f748ffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:52:23.264 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.250+0000 7f748ffff640 10 monclient(hunting): authenticate will time out at 1050.335414s 2024-03-29T11:52:23.264 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.250+0000 7f748dffb640 1 -- 172.21.15.98:0/2084943187 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 836839800 0 0) 0x7f74900aeda0 con 0x7f74900b4920 2024-03-29T11:52:23.264 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.250+0000 7f748dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:52:23.264 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.250+0000 7f748dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:23.264 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.250+0000 7f748dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:23.264 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.250+0000 7f748dffb640 10 monclient(hunting): my global_id is 4379 2024-03-29T11:52:23.264 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.250+0000 7f748dffb640 1 -- 172.21.15.98:0/2084943187 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7478003520 con 0x7f74900b4920 2024-03-29T11:52:23.264 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.251+0000 7f748dffb640 1 -- 172.21.15.98:0/2084943187 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 35231894 0 0) 0x7f7478003520 con 0x7f74900b4920 2024-03-29T11:52:23.264 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.251+0000 7f748dffb640 1 -- 172.21.15.98:0/2084943187 >> v1:172.21.15.121:6789/0 conn(0x7f74900b4200 legacy=0x7f74900b0fc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:52:23.264 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.251+0000 7f748dffb640 1 -- 172.21.15.98:0/2084943187 >> v1:172.21.15.98:6789/0 conn(0x7f749013f9a0 legacy=0x7f749014d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:52:23.265 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.251+0000 7f748dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:52:23.265 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.251+0000 7f748dffb640 1 monclient: found mon.noname-b 2024-03-29T11:52:23.265 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.251+0000 7f748dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:52:23.265 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.251+0000 7f748dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:52:23.265 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.251+0000 7f748dffb640 1 -- 172.21.15.98:0/2084943187 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74900af980 con 0x7f74900b4920 2024-03-29T11:52:23.265 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.251+0000 7f748dffb640 10 monclient: _finish_auth 0 2024-03-29T11:52:23.265 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.251+0000 7f748dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:52:23.265 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.251+0000 7f748dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:52:23.265 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.251+0000 7f748dffb640 1 -- 172.21.15.98:0/2084943187 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f7484001690 con 0x7f74900b4920 2024-03-29T11:52:23.265 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.251+0000 7f748dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:23.265 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.251+0000 7f748dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:52:23.265 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.251+0000 7f748dffb640 10 monclient: dump: 2024-03-29T11:52:23.265 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:52:23.265 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:23.266 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:23.266 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:23.266 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:23.266 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:23.266 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:52:23.266 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:52:23.266 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:52:23.266 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:23.266 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.251+0000 7f748ffff640 5 monclient: authenticate success, global_id 4379 2024-03-29T11:52:23.266 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.251+0000 7f748ffff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:52:23.266 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.251+0000 7f748dffb640 1 -- 172.21.15.98:0/2084943187 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7484001860 con 0x7f74900b4920 2024-03-29T11:52:23.266 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.251+0000 7f748dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:52:23.266 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.251+0000 7f748dffb640 1 -- 172.21.15.98:0/2084943187 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f7484001cb0 con 0x7f74900b4920 2024-03-29T11:52:23.266 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.251+0000 7f748dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:23.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.251+0000 7f748dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:52:23.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.251+0000 7f748dffb640 10 monclient: dump: 2024-03-29T11:52:23.267 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:52:23.267 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:23.267 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:23.267 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:23.267 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:23.267 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:23.267 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:52:23.267 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:52:23.267 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:52:23.267 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:23.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.251+0000 7f748ffff640 10 monclient: get_monmap_and_config success 2024-03-29T11:52:23.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.251+0000 7f748ffff640 4 set_mon_vals no callback set 2024-03-29T11:52:23.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.252+0000 7f748ffff640 10 monclient: shutdown 2024-03-29T11:52:23.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.252+0000 7f748ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:52:23.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.252+0000 7f748ffff640 1 -- 172.21.15.98:0/2084943187 >> v1:172.21.15.98:6790/0 conn(0x7f74900b4920 legacy=0x7f749013f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:23.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.253+0000 7f748ffff640 1 -- 172.21.15.98:0/2084943187 shutdown_connections 2024-03-29T11:52:23.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.253+0000 7f748ffff640 1 -- 172.21.15.98:0/2084943187 >> 172.21.15.98:0/2084943187 conn(0x7f749006d230 msgr2=0x7f7490066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:52:23.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.254+0000 7f748ffff640 1 -- 172.21.15.98:0/2084943187 shutdown_connections 2024-03-29T11:52:23.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.254+0000 7f748ffff640 1 -- 172.21.15.98:0/2084943187 wait complete. 2024-03-29T11:52:23.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.254+0000 7f748ffff640 10 monclient: build_initial_monmap 2024-03-29T11:52:23.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.254+0000 7f748ffff640 10 monclient: monmap: 2024-03-29T11:52:23.268 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:52:23.268 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:52:23.268 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:52:23.268 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:52:23.268 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:52:23.268 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:23.269 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:52:23.269 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:52:23.269 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:52:23.269 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:23.269 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.255+0000 7f748ffff640 1 librados: starting msgr at 2024-03-29T11:52:23.269 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.255+0000 7f748ffff640 1 librados: starting objecter 2024-03-29T11:52:23.269 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.255+0000 7f748ffff640 1 Processor -- start 2024-03-29T11:52:23.269 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.255+0000 7f748ffff640 1 -- start start 2024-03-29T11:52:23.269 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.255+0000 7f748ffff640 1 librados: setting wanted keys 2024-03-29T11:52:23.269 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.255+0000 7f748ffff640 1 librados: calling monclient init 2024-03-29T11:52:23.269 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.255+0000 7f748ffff640 10 monclient: init 2024-03-29T11:52:23.269 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.256+0000 7f748ffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:52:23.269 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.256+0000 7f748ffff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:52:23.269 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.256+0000 7f748ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f74900b4200 addr v1:172.21.15.98:6790/0 2024-03-29T11:52:23.274 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.256+0000 7f748ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f74900b4920 addr v1:172.21.15.121:6789/0 2024-03-29T11:52:23.274 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.257+0000 7f748ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f749013f9a0 addr v1:172.21.15.98:6789/0 2024-03-29T11:52:23.274 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.257+0000 7f748ffff640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7490141a40 con 0x7f749013f9a0 2024-03-29T11:52:23.274 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.257+0000 7f748ffff640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74901463d0 con 0x7f74900b4920 2024-03-29T11:52:23.274 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.257+0000 7f748ffff640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7490143150 con 0x7f74900b4200 2024-03-29T11:52:23.274 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.257+0000 7f748ffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:52:23.274 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.257+0000 7f748ffff640 10 monclient(hunting): authenticate will time out at 1050.342391s 2024-03-29T11:52:23.274 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.256+0000 7f748effd640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f74900b4200 0x7f74901d6470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:57140/0 (socket says 172.21.15.98:57140) 2024-03-29T11:52:23.274 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.257+0000 7f748effd640 1 -- 172.21.15.98:0/2753583341 learned_addr learned my addr 172.21.15.98:0/2753583341 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:52:23.274 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.257+0000 7f7477fff640 10 client.?.objecter ms_handle_connect 0x7f74900b4200 2024-03-29T11:52:23.274 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.257+0000 7f7477fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:52:23.274 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.257+0000 7f7477fff640 1 -- 172.21.15.98:0/2753583341 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1967303436 0 0) 0x7f7490143150 con 0x7f74900b4200 2024-03-29T11:52:23.274 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.257+0000 7f7477fff640 10 monclient(hunting): _negotiate 2024-03-29T11:52:23.275 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.257+0000 7f7477fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:23.275 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.257+0000 7f7477fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:23.275 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.257+0000 7f7477fff640 10 monclient(hunting): my global_id is 4382 2024-03-29T11:52:23.275 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.257+0000 7f7477fff640 1 -- 172.21.15.98:0/2753583341 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f747c004eb0 con 0x7f74900b4200 2024-03-29T11:52:23.275 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.258+0000 7f7477fff640 1 -- 172.21.15.98:0/2753583341 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2617009773 0 0) 0x7f747c004eb0 con 0x7f74900b4200 2024-03-29T11:52:23.275 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.258+0000 7f7477fff640 1 -- 172.21.15.98:0/2753583341 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7490143150 con 0x7f74900b4200 2024-03-29T11:52:23.275 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.258+0000 7f7477fff640 1 -- 172.21.15.98:0/2753583341 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f7484001b70 con 0x7f74900b4200 2024-03-29T11:52:23.275 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.258+0000 7f7477fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:23.275 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.258+0000 7f7477fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:52:23.275 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.258+0000 7f7477fff640 10 monclient(hunting): dump: 2024-03-29T11:52:23.275 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:52:23.275 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:23.275 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:23.275 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:23.275 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:23.276 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:23.276 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:52:23.276 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:52:23.276 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:52:23.276 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:23.276 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.258+0000 7f7477fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:52:23.276 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.258+0000 7f7477fff640 1 -- 172.21.15.98:0/2753583341 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2018858221 0 0) 0x7f7490143150 con 0x7f74900b4200 2024-03-29T11:52:23.276 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.258+0000 7f7477fff640 1 -- 172.21.15.98:0/2753583341 >> v1:172.21.15.121:6789/0 conn(0x7f74900b4920 legacy=0x7f74901d6b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:52:23.276 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.259+0000 7f7477fff640 1 -- 172.21.15.98:0/2753583341 >> v1:172.21.15.98:6789/0 conn(0x7f749013f9a0 legacy=0x7f74901dd530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:52:23.276 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.260+0000 7f7477fff640 10 monclient: _finish_hunting 0 2024-03-29T11:52:23.276 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.260+0000 7f7477fff640 1 monclient: found mon.c 2024-03-29T11:52:23.276 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.260+0000 7f7477fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:52:23.276 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.260+0000 7f7477fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:52:23.276 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.260+0000 7f7477fff640 1 -- 172.21.15.98:0/2753583341 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7490140b40 con 0x7f74900b4200 2024-03-29T11:52:23.276 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.260+0000 7f7477fff640 10 monclient: _finish_auth 0 2024-03-29T11:52:23.277 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.260+0000 7f7477fff640 10 monclient: _check_auth_tickets 2024-03-29T11:52:23.277 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.260+0000 7f7477fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:52:23.277 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.260+0000 7f748ffff640 5 monclient: authenticate success, global_id 4382 2024-03-29T11:52:23.277 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.260+0000 7f748ffff640 10 monclient: _renew_subs 2024-03-29T11:52:23.277 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.260+0000 7f748ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:52:23.277 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.260+0000 7f748ffff640 1 -- 172.21.15.98:0/2753583341 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7490146060 con 0x7f74900b4200 2024-03-29T11:52:23.277 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.260+0000 7f748ffff640 10 client.4382.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:52:23.277 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.260+0000 7f748ffff640 10 monclient: _renew_subs 2024-03-29T11:52:23.277 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.260+0000 7f748ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:52:23.277 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.260+0000 7f748ffff640 1 -- 172.21.15.98:0/2753583341 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f749013fe60 con 0x7f74900b4200 2024-03-29T11:52:23.277 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.260+0000 7f748ffff640 1 librados: init done 2024-03-29T11:52:23.277 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.260+0000 7f7477fff640 1 -- 172.21.15.98:0/2753583341 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7484004bf0 con 0x7f74900b4200 2024-03-29T11:52:23.277 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.260+0000 7f7477fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:52:23.277 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.260+0000 7f748d7fa640 4 set_mon_vals no callback set 2024-03-29T11:52:23.278 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.261+0000 7f7477fff640 1 -- 172.21.15.98:0/2753583341 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f7484004fc0 con 0x7f74900b4200 2024-03-29T11:52:23.278 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.261+0000 7f7477fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:23.278 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.261+0000 7f7477fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:52:23.278 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.261+0000 7f7477fff640 10 monclient: dump: 2024-03-29T11:52:23.278 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:52:23.278 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:23.278 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:23.278 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:23.278 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:23.278 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:23.278 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:52:23.278 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:52:23.278 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:52:23.279 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:23.279 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.263+0000 7f7477fff640 1 -- 172.21.15.98:0/2753583341 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f7484005120 con 0x7f74900b4200 2024-03-29T11:52:23.279 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.263+0000 7f7477fff640 1 -- 172.21.15.98:0/2753583341 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f7484084ce0 con 0x7f74900b4200 2024-03-29T11:52:23.279 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.263+0000 7f7477fff640 10 client.4382.objecter ms_dispatch 0x7f7490000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:52:23.279 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.263+0000 7f7477fff640 3 client.4382.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:52:23.279 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.263+0000 7f7477fff640 3 client.4382.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:52:23.279 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.263+0000 7f7477fff640 20 client.4382.objecter dump_active .. 0 homeless 2024-03-29T11:52:23.279 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.263+0000 7f748ffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:23.279 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.263+0000 7f748ffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:23.279 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.263+0000 7f748ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:52:23.279 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.263+0000 7f748ffff640 1 -- 172.21.15.98:0/2753583341 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7470005080 con 0x7f74900b4200 2024-03-29T11:52:23.279 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.264+0000 7f748e7fc640 10 monclient: get_auth_request con 0x7f747c06dd20 auth_method 0 2024-03-29T11:52:23.279 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.264+0000 7f7477fff640 10 client.4382.objecter ms_handle_connect 0x7f747c06dd20 2024-03-29T11:52:23.280 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.272+0000 7f7477fff640 1 -- 172.21.15.98:0/2753583341 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f74840505c0 con 0x7f74900b4200 2024-03-29T11:52:23.280 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.272+0000 7f7477fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:23.280 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.272+0000 7f7477fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:52:23.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.580+0000 7f748ffff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:23.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.580+0000 7f748ffff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:23.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.580+0000 7f748ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:52:23.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.580+0000 7f748ffff640 1 -- 172.21.15.98:0/2753583341 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7470005330 con 0x7f74900b4200 2024-03-29T11:52:23.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.581+0000 7f7477fff640 1 -- 172.21.15.98:0/2753583341 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f74840524f0 con 0x7f74900b4200 2024-03-29T11:52:23.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.581+0000 7f7477fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:23.589 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.581+0000 7f7477fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:52:23.589 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:52:23.589 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:52:23.591 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.583+0000 7f748ffff640 10 librados: watch_flush enter 2024-03-29T11:52:23.592 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.583+0000 7f748ffff640 10 librados: watch_flush exit 2024-03-29T11:52:23.592 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.583+0000 7f748ffff640 20 client.4382.objecter shutdown clearing up homeless session... 2024-03-29T11:52:23.592 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.584+0000 7f748ffff640 10 client.4382.objecter successfully canceled tick 2024-03-29T11:52:23.592 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.584+0000 7f748ffff640 1 -- 172.21.15.98:0/2753583341 >> v1:172.21.15.98:6816/4065186974 conn(0x7f747c06dd20 legacy=0x7f747c0701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:23.592 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.584+0000 7f748ffff640 10 monclient: shutdown 2024-03-29T11:52:23.592 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.584+0000 7f748ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:52:23.592 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.584+0000 7f748ffff640 1 -- 172.21.15.98:0/2753583341 >> v1:172.21.15.98:6790/0 conn(0x7f74900b4200 legacy=0x7f74901d6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:23.592 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.584+0000 7f748ffff640 1 -- 172.21.15.98:0/2753583341 shutdown_connections 2024-03-29T11:52:23.592 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.584+0000 7f748ffff640 1 -- 172.21.15.98:0/2753583341 >> 172.21.15.98:0/2753583341 conn(0x7f749006d230 msgr2=0x7f74900b6550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:52:23.592 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.584+0000 7f748ffff640 1 -- 172.21.15.98:0/2753583341 shutdown_connections 2024-03-29T11:52:23.592 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.584+0000 7f748ffff640 1 -- 172.21.15.98:0/2753583341 wait complete. 2024-03-29T11:52:23.592 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:23.584+0000 7f748ffff640 1 librados: shutdown 2024-03-29T11:52:23.607 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:52:30.413 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:52:30.558 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.551+0000 7f1ebb932640 10 monclient: get_monmap_and_config 2024-03-29T11:52:30.558 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.551+0000 7f1ebb932640 10 monclient: build_initial_monmap 2024-03-29T11:52:30.558 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.551+0000 7f1ebb932640 10 monclient: monmap: 2024-03-29T11:52:30.558 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:52:30.558 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:30.558 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:52:30.552484+0000 2024-03-29T11:52:30.559 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:52:30.552484+0000 2024-03-29T11:52:30.559 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:52:30.559 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:30.559 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:52:30.559 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:52:30.559 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:52:30.559 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:30.560 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.552+0000 7f1ebb932640 1 Processor -- start 2024-03-29T11:52:30.560 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.552+0000 7f1ebb932640 1 -- start start 2024-03-29T11:52:30.560 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.552+0000 7f1ebb932640 10 monclient: init 2024-03-29T11:52:30.560 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.553+0000 7f1ebb932640 10 monclient: _reopen_session rank -1 2024-03-29T11:52:30.560 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.553+0000 7f1ebb932640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:52:30.563 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.553+0000 7f1ebb932640 10 monclient(hunting): picked mon.noname-c con 0x7f1eb41457f0 addr v1:172.21.15.121:6789/0 2024-03-29T11:52:30.563 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.553+0000 7f1ebb932640 10 monclient(hunting): picked mon.noname-b con 0x7f1eb413f650 addr v1:172.21.15.98:6790/0 2024-03-29T11:52:30.563 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.553+0000 7f1ebb932640 10 monclient(hunting): picked mon.noname-a con 0x7f1eb40a9d70 addr v1:172.21.15.98:6789/0 2024-03-29T11:52:30.563 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.553+0000 7f1ebb932640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb40aaef0 con 0x7f1eb40a9d70 2024-03-29T11:52:30.563 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.554+0000 7f1ebb932640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb4145bd0 con 0x7f1eb41457f0 2024-03-29T11:52:30.563 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.554+0000 7f1ebb932640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb4145eb0 con 0x7f1eb413f650 2024-03-29T11:52:30.563 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.554+0000 7f1ebb932640 10 monclient(hunting): _renew_subs 2024-03-29T11:52:30.563 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.554+0000 7f1ebb932640 10 monclient(hunting): authenticate will time out at 1057.639083s 2024-03-29T11:52:30.564 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.557+0000 7f1eb8ecd640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f1eb413f650 0x7f1eb40b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:39055/0 (socket says 172.21.15.98:39055) 2024-03-29T11:52:30.564 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.557+0000 7f1eb8ecd640 1 -- 172.21.15.98:0/227483034 learned_addr learned my addr 172.21.15.98:0/227483034 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:52:30.566 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.558+0000 7f1eabfff640 1 -- 172.21.15.98:0/227483034 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1885095767 0 0) 0x7f1eb4145bd0 con 0x7f1eb41457f0 2024-03-29T11:52:30.566 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.558+0000 7f1eabfff640 10 monclient(hunting): _negotiate 2024-03-29T11:52:30.566 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.558+0000 7f1eabfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:30.566 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.558+0000 7f1eabfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:30.567 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.558+0000 7f1eabfff640 10 monclient(hunting): my global_id is 4393 2024-03-29T11:52:30.567 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.558+0000 7f1eabfff640 1 -- 172.21.15.98:0/227483034 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ea0003520 con 0x7f1eb41457f0 2024-03-29T11:52:30.567 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.559+0000 7f1eabfff640 1 -- 172.21.15.98:0/227483034 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2833642926 0 0) 0x7f1ea0003520 con 0x7f1eb41457f0 2024-03-29T11:52:30.567 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.559+0000 7f1eabfff640 1 -- 172.21.15.98:0/227483034 >> v1:172.21.15.98:6790/0 conn(0x7f1eb413f650 legacy=0x7f1eb40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:30.567 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.559+0000 7f1eabfff640 1 -- 172.21.15.98:0/227483034 >> v1:172.21.15.98:6789/0 conn(0x7f1eb40a9d70 legacy=0x7f1eb414d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:52:30.568 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.560+0000 7f1eabfff640 10 monclient: _finish_hunting 0 2024-03-29T11:52:30.569 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.560+0000 7f1eabfff640 1 monclient: found mon.noname-c 2024-03-29T11:52:30.569 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.560+0000 7f1eabfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:52:30.569 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.560+0000 7f1eabfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:52:30.569 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.560+0000 7f1eabfff640 1 -- 172.21.15.98:0/227483034 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb4146340 con 0x7f1eb41457f0 2024-03-29T11:52:30.569 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.560+0000 7f1eabfff640 10 monclient: _finish_auth 0 2024-03-29T11:52:30.569 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.560+0000 7f1eabfff640 10 monclient: _check_auth_tickets 2024-03-29T11:52:30.569 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.560+0000 7f1eabfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:52:30.569 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.560+0000 7f1eabfff640 1 -- 172.21.15.98:0/227483034 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f1eb00015c0 con 0x7f1eb41457f0 2024-03-29T11:52:30.569 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.560+0000 7f1eabfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:30.569 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.560+0000 7f1eabfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:52:30.569 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.560+0000 7f1eabfff640 10 monclient: dump: 2024-03-29T11:52:30.569 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:52:30.569 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:30.569 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:30.570 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:30.570 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:30.570 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:30.570 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:52:30.570 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:52:30.570 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:52:30.570 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:30.570 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.560+0000 7f1ebb932640 5 monclient: authenticate success, global_id 4393 2024-03-29T11:52:30.570 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.560+0000 7f1ebb932640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:52:30.571 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.562+0000 7f1eabfff640 1 -- 172.21.15.98:0/227483034 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1eb0001790 con 0x7f1eb41457f0 2024-03-29T11:52:30.571 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.562+0000 7f1eabfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:52:30.571 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.562+0000 7f1eabfff640 1 -- 172.21.15.98:0/227483034 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f1eb0001be0 con 0x7f1eb41457f0 2024-03-29T11:52:30.571 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.562+0000 7f1eabfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:30.571 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.562+0000 7f1eabfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:52:30.571 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.562+0000 7f1eabfff640 10 monclient: dump: 2024-03-29T11:52:30.571 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:52:30.571 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:30.571 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:30.571 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:30.571 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:30.571 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:30.572 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:52:30.572 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:52:30.572 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:52:30.572 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:30.572 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.562+0000 7f1ebb932640 10 monclient: get_monmap_and_config success 2024-03-29T11:52:30.572 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.562+0000 7f1ebb932640 4 set_mon_vals no callback set 2024-03-29T11:52:30.572 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.563+0000 7f1ebb932640 10 monclient: shutdown 2024-03-29T11:52:30.572 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.563+0000 7f1ebb932640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:52:30.572 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.563+0000 7f1ebb932640 1 -- 172.21.15.98:0/227483034 >> v1:172.21.15.121:6789/0 conn(0x7f1eb41457f0 legacy=0x7f1eb413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:30.572 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.564+0000 7f1ebb932640 1 -- 172.21.15.98:0/227483034 shutdown_connections 2024-03-29T11:52:30.572 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.564+0000 7f1ebb932640 1 -- 172.21.15.98:0/227483034 >> 172.21.15.98:0/227483034 conn(0x7f1eb4066a00 msgr2=0x7f1eb4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:52:30.573 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.565+0000 7f1ebb932640 1 -- 172.21.15.98:0/227483034 shutdown_connections 2024-03-29T11:52:30.573 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.565+0000 7f1ebb932640 1 -- 172.21.15.98:0/227483034 wait complete. 2024-03-29T11:52:30.575 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.566+0000 7f1ebb932640 10 monclient: build_initial_monmap 2024-03-29T11:52:30.575 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.566+0000 7f1ebb932640 10 monclient: monmap: 2024-03-29T11:52:30.575 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:52:30.575 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:52:30.575 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:52:30.575 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:52:30.575 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:52:30.575 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:30.575 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:52:30.575 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:52:30.576 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:52:30.576 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:30.576 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.566+0000 7f1ebb932640 1 librados: starting msgr at 2024-03-29T11:52:30.576 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.566+0000 7f1ebb932640 1 librados: starting objecter 2024-03-29T11:52:30.576 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.566+0000 7f1ebb932640 1 Processor -- start 2024-03-29T11:52:30.576 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.566+0000 7f1ebb932640 1 -- start start 2024-03-29T11:52:30.576 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.566+0000 7f1ebb932640 1 librados: setting wanted keys 2024-03-29T11:52:30.576 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.566+0000 7f1ebb932640 1 librados: calling monclient init 2024-03-29T11:52:30.576 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.566+0000 7f1ebb932640 10 monclient: init 2024-03-29T11:52:30.576 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.567+0000 7f1ebb932640 10 monclient: _reopen_session rank -1 2024-03-29T11:52:30.576 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.567+0000 7f1ebb932640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:52:30.576 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.567+0000 7f1ebb932640 10 monclient(hunting): picked mon.noname-a con 0x7f1eb40a9d70 addr v1:172.21.15.98:6789/0 2024-03-29T11:52:30.576 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.567+0000 7f1ebb932640 10 monclient(hunting): picked mon.noname-c con 0x7f1eb413f650 addr v1:172.21.15.98:6790/0 2024-03-29T11:52:30.576 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.568+0000 7f1ebb932640 10 monclient(hunting): picked mon.noname-b con 0x7f1eb41457f0 addr v1:172.21.15.121:6789/0 2024-03-29T11:52:30.576 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.568+0000 7f1ebb932640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb4135d90 con 0x7f1eb40a9d70 2024-03-29T11:52:30.577 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.568+0000 7f1ebb932640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb4140db0 con 0x7f1eb41457f0 2024-03-29T11:52:30.577 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.568+0000 7f1ebb932640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb4136000 con 0x7f1eb413f650 2024-03-29T11:52:30.577 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.568+0000 7f1ebb932640 10 monclient(hunting): _renew_subs 2024-03-29T11:52:30.577 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.568+0000 7f1ebb932640 10 monclient(hunting): authenticate will time out at 1057.653344s 2024-03-29T11:52:30.583 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.572+0000 7f1eb8ecd640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f1eb413f650 0x7f1eb41e34a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:39159/0 (socket says 172.21.15.98:39159) 2024-03-29T11:52:30.584 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.572+0000 7f1eb8ecd640 1 -- 172.21.15.98:0/856901378 learned_addr learned my addr 172.21.15.98:0/856901378 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:52:30.584 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.573+0000 7f1ea9ffb640 10 client.?.objecter ms_handle_connect 0x7f1eb413f650 2024-03-29T11:52:30.584 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.573+0000 7f1ea9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:52:30.584 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.573+0000 7f1ea9ffb640 10 client.?.objecter ms_handle_connect 0x7f1eb40a9d70 2024-03-29T11:52:30.584 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.573+0000 7f1ea9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:52:30.584 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.573+0000 7f1ea9ffb640 1 -- 172.21.15.98:0/856901378 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 506115113 0 0) 0x7f1eb4136000 con 0x7f1eb413f650 2024-03-29T11:52:30.584 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.573+0000 7f1ea9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:52:30.584 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.573+0000 7f1ea9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:30.584 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.573+0000 7f1ea9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:30.584 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.573+0000 7f1ea9ffb640 10 monclient(hunting): my global_id is 4388 2024-03-29T11:52:30.584 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.573+0000 7f1ea9ffb640 1 -- 172.21.15.98:0/856901378 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e98004eb0 con 0x7f1eb413f650 2024-03-29T11:52:30.584 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.573+0000 7f1ea9ffb640 1 -- 172.21.15.98:0/856901378 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2989041264 0 0) 0x7f1eb4135d90 con 0x7f1eb40a9d70 2024-03-29T11:52:30.584 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.573+0000 7f1ea9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:52:30.584 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.573+0000 7f1ea9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:30.584 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.573+0000 7f1ea9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:30.585 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.573+0000 7f1ea9ffb640 10 monclient(hunting): my global_id is 4356 2024-03-29T11:52:30.585 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.573+0000 7f1ea9ffb640 1 -- 172.21.15.98:0/856901378 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1eb4136000 con 0x7f1eb40a9d70 2024-03-29T11:52:30.585 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.573+0000 7f1ea9ffb640 1 -- 172.21.15.98:0/856901378 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2738969055 0 0) 0x7f1e98004eb0 con 0x7f1eb413f650 2024-03-29T11:52:30.585 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.573+0000 7f1ea9ffb640 1 -- 172.21.15.98:0/856901378 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1eb4135d90 con 0x7f1eb413f650 2024-03-29T11:52:30.585 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.573+0000 7f1ea9ffb640 1 -- 172.21.15.98:0/856901378 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f1ea4001930 con 0x7f1eb413f650 2024-03-29T11:52:30.585 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.573+0000 7f1ea9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:30.585 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.573+0000 7f1ea9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:52:30.585 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.573+0000 7f1ea9ffb640 10 monclient(hunting): dump: 2024-03-29T11:52:30.585 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:52:30.585 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:30.585 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:30.585 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:30.585 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:30.585 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:30.586 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:52:30.586 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:52:30.586 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:52:30.586 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:30.586 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.573+0000 7f1ea9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:52:30.586 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.574+0000 7f1ea9ffb640 1 -- 172.21.15.98:0/856901378 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3728506201 0 0) 0x7f1eb4136000 con 0x7f1eb40a9d70 2024-03-29T11:52:30.586 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.574+0000 7f1ea9ffb640 1 -- 172.21.15.98:0/856901378 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e98004eb0 con 0x7f1eb40a9d70 2024-03-29T11:52:30.586 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.574+0000 7f1ea9ffb640 1 -- 172.21.15.98:0/856901378 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2684638967 0 0) 0x7f1eb4135d90 con 0x7f1eb413f650 2024-03-29T11:52:30.586 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.574+0000 7f1ea9ffb640 1 -- 172.21.15.98:0/856901378 >> v1:172.21.15.121:6789/0 conn(0x7f1eb41457f0 legacy=0x7f1eb41e6bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:52:30.586 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.574+0000 7f1ea9ffb640 1 -- 172.21.15.98:0/856901378 >> v1:172.21.15.98:6789/0 conn(0x7f1eb40a9d70 legacy=0x7f1eb4136790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:30.586 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.574+0000 7f1ea9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:52:30.586 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.574+0000 7f1ea9ffb640 1 monclient: found mon.c 2024-03-29T11:52:30.586 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.574+0000 7f1ea9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:52:30.586 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.575+0000 7f1ea9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:52:30.586 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.575+0000 7f1ea9ffb640 1 -- 172.21.15.98:0/856901378 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb41346b0 con 0x7f1eb413f650 2024-03-29T11:52:30.586 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.575+0000 7f1ea9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:52:30.586 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.575+0000 7f1ea9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:52:30.587 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.575+0000 7f1ea9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:52:30.587 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.575+0000 7f1ebb932640 5 monclient: authenticate success, global_id 4388 2024-03-29T11:52:30.587 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.575+0000 7f1ebb932640 10 monclient: _renew_subs 2024-03-29T11:52:30.587 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.575+0000 7f1ebb932640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:52:30.587 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.575+0000 7f1ebb932640 1 -- 172.21.15.98:0/856901378 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eb4142ca0 con 0x7f1eb413f650 2024-03-29T11:52:30.587 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.575+0000 7f1ea9ffb640 1 -- 172.21.15.98:0/856901378 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ea40047e0 con 0x7f1eb413f650 2024-03-29T11:52:30.587 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.575+0000 7f1ea9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:52:30.587 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.575+0000 7f1ea9ffb640 1 -- 172.21.15.98:0/856901378 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f1ea4004b70 con 0x7f1eb413f650 2024-03-29T11:52:30.587 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.575+0000 7f1ea9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:30.587 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.575+0000 7f1ea9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:52:30.587 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.575+0000 7f1ea9ffb640 10 monclient: dump: 2024-03-29T11:52:30.587 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:52:30.587 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:30.587 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:30.587 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:30.587 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:30.588 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:30.588 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:52:30.588 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:52:30.588 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:52:30.588 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:30.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.575+0000 7f1eaaffd640 4 set_mon_vals no callback set 2024-03-29T11:52:30.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.575+0000 7f1ebb932640 10 client.4388.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:52:30.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.575+0000 7f1ebb932640 10 monclient: _renew_subs 2024-03-29T11:52:30.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.575+0000 7f1ebb932640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:52:30.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.575+0000 7f1ebb932640 1 -- 172.21.15.98:0/856901378 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eb4134bc0 con 0x7f1eb413f650 2024-03-29T11:52:30.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.575+0000 7f1ebb932640 1 librados: init done 2024-03-29T11:52:30.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.576+0000 7f1e9f7fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:30.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.576+0000 7f1e9f7fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:30.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.576+0000 7f1e9f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:52:30.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.576+0000 7f1e9f7fe640 1 -- 172.21.15.98:0/856901378 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e8c005080 con 0x7f1eb413f650 2024-03-29T11:52:30.589 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.577+0000 7f1ea9ffb640 1 -- 172.21.15.98:0/856901378 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f1ea4004cd0 con 0x7f1eb413f650 2024-03-29T11:52:30.589 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.577+0000 7f1eb96ce640 10 monclient: get_auth_request con 0x7f1e9806dd20 auth_method 0 2024-03-29T11:52:30.589 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.578+0000 7f1ea9ffb640 10 client.4388.objecter ms_handle_connect 0x7f1e9806dd20 2024-03-29T11:52:30.589 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.578+0000 7f1ea9ffb640 1 -- 172.21.15.98:0/856901378 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f1ea4054860 con 0x7f1eb413f650 2024-03-29T11:52:30.589 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.578+0000 7f1ea9ffb640 10 client.4388.objecter ms_dispatch 0x7f1eb4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:52:30.590 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.578+0000 7f1ea9ffb640 3 client.4388.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:52:30.590 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.578+0000 7f1ea9ffb640 3 client.4388.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:52:30.590 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.578+0000 7f1ea9ffb640 20 client.4388.objecter dump_active .. 0 homeless 2024-03-29T11:52:30.595 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.587+0000 7f1ea9ffb640 1 -- 172.21.15.98:0/856901378 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1ea4050b00 con 0x7f1eb413f650 2024-03-29T11:52:30.595 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.587+0000 7f1ea9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:30.595 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.587+0000 7f1ea9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:52:30.926 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.918+0000 7f1e9f7fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:30.926 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.918+0000 7f1e9f7fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:30.926 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.918+0000 7f1e9f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:52:30.926 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.918+0000 7f1e9f7fe640 1 -- 172.21.15.98:0/856901378 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e8c005330 con 0x7f1eb413f650 2024-03-29T11:52:30.927 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.920+0000 7f1ea9ffb640 1 -- 172.21.15.98:0/856901378 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f1ea4050b00 con 0x7f1eb413f650 2024-03-29T11:52:30.927 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.920+0000 7f1ea9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:30.927 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.920+0000 7f1ea9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:52:30.928 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:52:30.928 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:52:30.931 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.923+0000 7f1e9f7fe640 10 librados: watch_flush enter 2024-03-29T11:52:30.931 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.924+0000 7f1e9f7fe640 10 librados: watch_flush exit 2024-03-29T11:52:30.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.924+0000 7f1e9f7fe640 20 client.4388.objecter shutdown clearing up homeless session... 2024-03-29T11:52:30.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.924+0000 7f1e9f7fe640 10 client.4388.objecter successfully canceled tick 2024-03-29T11:52:30.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.924+0000 7f1e9f7fe640 1 -- 172.21.15.98:0/856901378 >> v1:172.21.15.98:6816/4065186974 conn(0x7f1e9806dd20 legacy=0x7f1e980701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:30.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.924+0000 7f1e9f7fe640 10 monclient: shutdown 2024-03-29T11:52:30.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.924+0000 7f1e9f7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:52:30.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.924+0000 7f1e9f7fe640 1 -- 172.21.15.98:0/856901378 >> v1:172.21.15.98:6790/0 conn(0x7f1eb413f650 legacy=0x7f1eb41e34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:30.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.924+0000 7f1e9f7fe640 1 -- 172.21.15.98:0/856901378 shutdown_connections 2024-03-29T11:52:30.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.924+0000 7f1e9f7fe640 1 -- 172.21.15.98:0/856901378 >> 172.21.15.98:0/856901378 conn(0x7f1eb4066a00 msgr2=0x7f1eb413ac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:52:30.933 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.926+0000 7f1e9f7fe640 1 -- 172.21.15.98:0/856901378 shutdown_connections 2024-03-29T11:52:30.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.928+0000 7f1e9f7fe640 1 -- 172.21.15.98:0/856901378 wait complete. 2024-03-29T11:52:30.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:30.929+0000 7f1e9f7fe640 1 librados: shutdown 2024-03-29T11:52:30.957 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:52:37.764 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:52:37.922 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.914+0000 7f71138c1640 10 monclient: get_monmap_and_config 2024-03-29T11:52:37.922 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.914+0000 7f71138c1640 10 monclient: build_initial_monmap 2024-03-29T11:52:37.922 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.914+0000 7f71138c1640 10 monclient: monmap: 2024-03-29T11:52:37.922 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:52:37.922 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:37.922 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:52:37.915330+0000 2024-03-29T11:52:37.922 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:52:37.915330+0000 2024-03-29T11:52:37.922 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:52:37.922 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:37.922 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:52:37.922 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:52:37.922 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:52:37.923 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:37.923 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.915+0000 7f71138c1640 1 Processor -- start 2024-03-29T11:52:37.924 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.915+0000 7f71138c1640 1 -- start start 2024-03-29T11:52:37.924 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.915+0000 7f71138c1640 10 monclient: init 2024-03-29T11:52:37.924 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.916+0000 7f71138c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:52:37.924 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.916+0000 7f71138c1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:52:37.926 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.917+0000 7f71138c1640 10 monclient(hunting): picked mon.noname-b con 0x7f710c0aa990 addr v1:172.21.15.98:6790/0 2024-03-29T11:52:37.927 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.919+0000 7f71138c1640 10 monclient(hunting): picked mon.noname-c con 0x7f710c0a9da0 addr v1:172.21.15.121:6789/0 2024-03-29T11:52:37.927 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.919+0000 7f71138c1640 10 monclient(hunting): picked mon.noname-a con 0x7f710c0adbb0 addr v1:172.21.15.98:6789/0 2024-03-29T11:52:37.927 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.919+0000 7f71138c1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f710c0ad780 con 0x7f710c0adbb0 2024-03-29T11:52:37.927 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.919+0000 7f71138c1640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f710c0ab090 con 0x7f710c0a9da0 2024-03-29T11:52:37.927 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.919+0000 7f71138c1640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f710c0aad70 con 0x7f710c0aa990 2024-03-29T11:52:37.927 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.919+0000 7f71138c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:52:37.927 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.919+0000 7f71138c1640 10 monclient(hunting): authenticate will time out at 1065.004282s 2024-03-29T11:52:37.927 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.919+0000 7f7110e5c640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f710c0a9da0 0x7f710c0ac960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:48632/0 (socket says 172.21.15.98:48632) 2024-03-29T11:52:37.927 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.919+0000 7f7110e5c640 1 -- 172.21.15.98:0/1486425375 learned_addr learned my addr 172.21.15.98:0/1486425375 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:52:37.928 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.920+0000 7f7103fff640 1 -- 172.21.15.98:0/1486425375 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4046329391 0 0) 0x7f710c0ab090 con 0x7f710c0a9da0 2024-03-29T11:52:37.928 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.920+0000 7f7103fff640 10 monclient(hunting): _negotiate 2024-03-29T11:52:37.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.920+0000 7f7103fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:37.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.920+0000 7f7103fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:37.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.920+0000 7f7103fff640 10 monclient(hunting): my global_id is 4396 2024-03-29T11:52:37.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.920+0000 7f7103fff640 1 -- 172.21.15.98:0/1486425375 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70f8003520 con 0x7f710c0a9da0 2024-03-29T11:52:37.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.920+0000 7f7103fff640 1 -- 172.21.15.98:0/1486425375 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1620148610 0 0) 0x7f70f8003520 con 0x7f710c0a9da0 2024-03-29T11:52:37.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.920+0000 7f7103fff640 1 -- 172.21.15.98:0/1486425375 >> v1:172.21.15.98:6790/0 conn(0x7f710c0aa990 legacy=0x7f710c13b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:52:37.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.920+0000 7f7103fff640 1 -- 172.21.15.98:0/1486425375 >> v1:172.21.15.98:6789/0 conn(0x7f710c0adbb0 legacy=0x7f710c0ad070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:52:37.931 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.922+0000 7f7103fff640 10 monclient: _finish_hunting 0 2024-03-29T11:52:37.931 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.922+0000 7f7103fff640 1 monclient: found mon.noname-c 2024-03-29T11:52:37.931 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.922+0000 7f7103fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:52:37.931 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.923+0000 7f7103fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:52:37.931 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.923+0000 7f7103fff640 1 -- 172.21.15.98:0/1486425375 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f710c13b810 con 0x7f710c0a9da0 2024-03-29T11:52:37.931 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.923+0000 7f7103fff640 10 monclient: _finish_auth 0 2024-03-29T11:52:37.931 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.923+0000 7f7103fff640 10 monclient: _check_auth_tickets 2024-03-29T11:52:37.931 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.923+0000 7f7103fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:52:37.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.923+0000 7f7103fff640 1 -- 172.21.15.98:0/1486425375 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f70fc001690 con 0x7f710c0a9da0 2024-03-29T11:52:37.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.923+0000 7f7103fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:37.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.923+0000 7f7103fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:52:37.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.923+0000 7f7103fff640 10 monclient: dump: 2024-03-29T11:52:37.932 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:52:37.932 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:37.932 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:37.932 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:37.932 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:37.932 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:37.932 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:52:37.932 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:52:37.932 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:52:37.932 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:37.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.923+0000 7f71138c1640 5 monclient: authenticate success, global_id 4396 2024-03-29T11:52:37.933 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.923+0000 7f71138c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:52:37.933 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.924+0000 7f7103fff640 1 -- 172.21.15.98:0/1486425375 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70fc001860 con 0x7f710c0a9da0 2024-03-29T11:52:37.933 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.924+0000 7f7103fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:52:37.933 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.924+0000 7f7103fff640 1 -- 172.21.15.98:0/1486425375 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f70fc001cb0 con 0x7f710c0a9da0 2024-03-29T11:52:37.933 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.924+0000 7f7103fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:37.933 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.924+0000 7f7103fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:52:37.933 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.924+0000 7f7103fff640 10 monclient: dump: 2024-03-29T11:52:37.933 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:52:37.933 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:37.933 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:37.933 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:37.933 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:37.933 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:37.933 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:52:37.933 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:52:37.933 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:52:37.934 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:37.935 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.925+0000 7f71138c1640 10 monclient: get_monmap_and_config success 2024-03-29T11:52:37.935 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.925+0000 7f71138c1640 4 set_mon_vals no callback set 2024-03-29T11:52:37.935 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.926+0000 7f71138c1640 10 monclient: shutdown 2024-03-29T11:52:37.935 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.926+0000 7f71138c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:52:37.935 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.926+0000 7f71138c1640 1 -- 172.21.15.98:0/1486425375 >> v1:172.21.15.121:6789/0 conn(0x7f710c0a9da0 legacy=0x7f710c0ac960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:37.935 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.926+0000 7f71138c1640 1 -- 172.21.15.98:0/1486425375 shutdown_connections 2024-03-29T11:52:37.935 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.926+0000 7f71138c1640 1 -- 172.21.15.98:0/1486425375 >> 172.21.15.98:0/1486425375 conn(0x7f710c0b3c80 msgr2=0x7f710c0b4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:52:37.935 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.927+0000 7f71138c1640 1 -- 172.21.15.98:0/1486425375 shutdown_connections 2024-03-29T11:52:37.935 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.927+0000 7f71138c1640 1 -- 172.21.15.98:0/1486425375 wait complete. 2024-03-29T11:52:37.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.928+0000 7f71138c1640 10 monclient: build_initial_monmap 2024-03-29T11:52:37.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.928+0000 7f71138c1640 10 monclient: monmap: 2024-03-29T11:52:37.936 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:52:37.936 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:52:37.936 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:52:37.936 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:52:37.936 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:52:37.936 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:37.936 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:52:37.936 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:52:37.937 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:52:37.937 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:37.937 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.929+0000 7f71138c1640 1 librados: starting msgr at 2024-03-29T11:52:37.937 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.929+0000 7f71138c1640 1 librados: starting objecter 2024-03-29T11:52:37.937 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.929+0000 7f71138c1640 1 Processor -- start 2024-03-29T11:52:37.938 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.929+0000 7f71138c1640 1 -- start start 2024-03-29T11:52:37.938 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.929+0000 7f71138c1640 1 librados: setting wanted keys 2024-03-29T11:52:37.938 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.929+0000 7f71138c1640 1 librados: calling monclient init 2024-03-29T11:52:37.938 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.929+0000 7f71138c1640 10 monclient: init 2024-03-29T11:52:37.938 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.930+0000 7f71138c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:52:37.938 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.930+0000 7f71138c1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:52:37.938 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.931+0000 7f71138c1640 10 monclient(hunting): picked mon.noname-b con 0x7f710c0a9da0 addr v1:172.21.15.121:6789/0 2024-03-29T11:52:37.938 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.931+0000 7f71138c1640 10 monclient(hunting): picked mon.noname-c con 0x7f710c0aa990 addr v1:172.21.15.98:6790/0 2024-03-29T11:52:37.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.931+0000 7f71138c1640 10 monclient(hunting): picked mon.noname-a con 0x7f710c0adbb0 addr v1:172.21.15.98:6789/0 2024-03-29T11:52:37.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.931+0000 7f71138c1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f710c06dca0 con 0x7f710c0adbb0 2024-03-29T11:52:37.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.931+0000 7f71138c1640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f710c14da30 con 0x7f710c0a9da0 2024-03-29T11:52:37.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.931+0000 7f71138c1640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f710c135420 con 0x7f710c0aa990 2024-03-29T11:52:37.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.931+0000 7f71138c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:52:37.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.931+0000 7f71138c1640 10 monclient(hunting): authenticate will time out at 1065.016819s 2024-03-29T11:52:37.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.935+0000 7f711165d640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f710c0a9da0 0x7f710c1d8260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:50048/0 (socket says 172.21.15.98:50048) 2024-03-29T11:52:37.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.935+0000 7f7111e5e640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f710c0adbb0 0x7f710c134d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54939/0 (socket says 172.21.15.98:54939) 2024-03-29T11:52:37.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.935+0000 7f7111e5e640 1 -- 172.21.15.98:0/132431575 learned_addr learned my addr 172.21.15.98:0/132431575 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:52:37.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.935+0000 7f7101ffb640 10 client.?.objecter ms_handle_connect 0x7f710c0adbb0 2024-03-29T11:52:37.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.935+0000 7f7101ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:52:37.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.936+0000 7f7101ffb640 10 client.?.objecter ms_handle_connect 0x7f710c0a9da0 2024-03-29T11:52:37.945 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.936+0000 7f7101ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:52:37.945 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.936+0000 7f7101ffb640 1 -- 172.21.15.98:0/132431575 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 340524316 0 0) 0x7f710c06dca0 con 0x7f710c0adbb0 2024-03-29T11:52:37.945 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.936+0000 7f7101ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:52:37.945 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.936+0000 7f7101ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:37.945 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.936+0000 7f7101ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:37.945 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.936+0000 7f7101ffb640 10 monclient(hunting): my global_id is 4359 2024-03-29T11:52:37.945 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.936+0000 7f7101ffb640 1 -- 172.21.15.98:0/132431575 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70f0004eb0 con 0x7f710c0adbb0 2024-03-29T11:52:37.945 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.936+0000 7f7101ffb640 1 -- 172.21.15.98:0/132431575 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2713033234 0 0) 0x7f710c14da30 con 0x7f710c0a9da0 2024-03-29T11:52:37.945 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.936+0000 7f7101ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:52:37.945 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.936+0000 7f7101ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:37.945 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.936+0000 7f7101ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:37.945 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.936+0000 7f7101ffb640 10 monclient(hunting): my global_id is 4399 2024-03-29T11:52:37.945 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.936+0000 7f7101ffb640 1 -- 172.21.15.98:0/132431575 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f710c06dca0 con 0x7f710c0a9da0 2024-03-29T11:52:37.946 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.938+0000 7f7101ffb640 1 -- 172.21.15.98:0/132431575 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3643986637 0 0) 0x7f710c06dca0 con 0x7f710c0a9da0 2024-03-29T11:52:37.946 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.938+0000 7f7101ffb640 1 -- 172.21.15.98:0/132431575 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f710c14da30 con 0x7f710c0a9da0 2024-03-29T11:52:37.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.938+0000 7f7101ffb640 1 -- 172.21.15.98:0/132431575 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f7108001860 con 0x7f710c0a9da0 2024-03-29T11:52:37.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.938+0000 7f7101ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:37.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.938+0000 7f7101ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:52:37.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.938+0000 7f7101ffb640 10 monclient(hunting): dump: 2024-03-29T11:52:37.947 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:52:37.947 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:37.947 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:37.947 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:37.947 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:37.947 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:37.947 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:52:37.947 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:52:37.947 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:52:37.947 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:37.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.938+0000 7f7101ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:52:37.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.940+0000 7f7101ffb640 10 client.?.objecter ms_handle_connect 0x7f710c0aa990 2024-03-29T11:52:37.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.940+0000 7f7101ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:52:37.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.940+0000 7f7101ffb640 1 -- 172.21.15.98:0/132431575 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 623866614 0 0) 0x7f710c135420 con 0x7f710c0aa990 2024-03-29T11:52:37.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.940+0000 7f7101ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:52:37.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.940+0000 7f7101ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:37.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.940+0000 7f7101ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:37.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.940+0000 7f7101ffb640 10 monclient(hunting): my global_id is 4391 2024-03-29T11:52:37.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.940+0000 7f7101ffb640 1 -- 172.21.15.98:0/132431575 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f710c06dca0 con 0x7f710c0aa990 2024-03-29T11:52:37.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.941+0000 7f7101ffb640 1 -- 172.21.15.98:0/132431575 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 317186536 0 0) 0x7f710c14da30 con 0x7f710c0a9da0 2024-03-29T11:52:37.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.941+0000 7f7101ffb640 1 -- 172.21.15.98:0/132431575 >> v1:172.21.15.98:6790/0 conn(0x7f710c0aa990 legacy=0x7f710c134600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:37.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.941+0000 7f7101ffb640 1 -- 172.21.15.98:0/132431575 >> v1:172.21.15.98:6789/0 conn(0x7f710c0adbb0 legacy=0x7f710c134d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:37.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.941+0000 7f7101ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:52:37.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.941+0000 7f7101ffb640 1 monclient: found mon.b 2024-03-29T11:52:37.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.941+0000 7f7101ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:52:37.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.941+0000 7f7101ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:52:37.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.941+0000 7f7101ffb640 1 -- 172.21.15.98:0/132431575 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f710c14ee10 con 0x7f710c0a9da0 2024-03-29T11:52:37.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.941+0000 7f7101ffb640 10 monclient: _finish_auth 0 2024-03-29T11:52:37.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.941+0000 7f7101ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:52:37.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.941+0000 7f7101ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:52:37.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.942+0000 7f7101ffb640 1 -- 172.21.15.98:0/132431575 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7108001d70 con 0x7f710c0a9da0 2024-03-29T11:52:37.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.942+0000 7f7101ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:52:37.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.942+0000 7f7101ffb640 1 -- 172.21.15.98:0/132431575 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f7108003790 con 0x7f710c0a9da0 2024-03-29T11:52:37.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.942+0000 7f7101ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:37.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.942+0000 7f7101ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:52:37.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.942+0000 7f7101ffb640 10 monclient: dump: 2024-03-29T11:52:37.950 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:52:37.950 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:37.950 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:37.950 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:37.950 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:37.951 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:37.951 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:52:37.951 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:52:37.951 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:52:37.951 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:37.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.942+0000 7f71037fe640 4 set_mon_vals no callback set 2024-03-29T11:52:37.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.942+0000 7f71138c1640 5 monclient: authenticate success, global_id 4399 2024-03-29T11:52:37.952 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.942+0000 7f71138c1640 10 monclient: _renew_subs 2024-03-29T11:52:37.952 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.942+0000 7f71138c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:52:37.952 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.942+0000 7f71138c1640 1 -- 172.21.15.98:0/132431575 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f710c14d550 con 0x7f710c0a9da0 2024-03-29T11:52:37.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.944+0000 7f71138c1640 10 client.4399.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:52:37.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.944+0000 7f71138c1640 10 monclient: _renew_subs 2024-03-29T11:52:37.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.944+0000 7f71138c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:52:37.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.944+0000 7f71138c1640 1 -- 172.21.15.98:0/132431575 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f710c14e0a0 con 0x7f710c0a9da0 2024-03-29T11:52:37.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.944+0000 7f71138c1640 1 librados: init done 2024-03-29T11:52:37.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.945+0000 7f7101ffb640 1 -- 172.21.15.98:0/132431575 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f71080038f0 con 0x7f710c0a9da0 2024-03-29T11:52:37.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.946+0000 7f71138c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:37.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.946+0000 7f71138c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:37.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.946+0000 7f71138c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:52:37.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.946+0000 7f71138c1640 1 -- 172.21.15.98:0/132431575 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f710c14a690 con 0x7f710c0a9da0 2024-03-29T11:52:37.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.946+0000 7f7101ffb640 1 -- 172.21.15.98:0/132431575 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f71080547d0 con 0x7f710c0a9da0 2024-03-29T11:52:37.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.946+0000 7f7101ffb640 10 client.4399.objecter ms_dispatch 0x7f710c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:52:37.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.946+0000 7f7101ffb640 3 client.4399.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:52:37.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.946+0000 7f7101ffb640 3 client.4399.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:52:37.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.946+0000 7f7101ffb640 20 client.4399.objecter dump_active .. 0 homeless 2024-03-29T11:52:37.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.946+0000 7f7110e5c640 10 monclient: get_auth_request con 0x7f70f0076630 auth_method 0 2024-03-29T11:52:37.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.946+0000 7f7101ffb640 10 client.4399.objecter ms_handle_connect 0x7f70f0076630 2024-03-29T11:52:37.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.952+0000 7f7101ffb640 1 -- 172.21.15.98:0/132431575 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7108050a70 con 0x7f710c0a9da0 2024-03-29T11:52:37.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.952+0000 7f7101ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:37.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:37.952+0000 7f7101ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:52:38.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:38.287+0000 7f71138c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:38.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:38.287+0000 7f71138c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:38.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:38.287+0000 7f71138c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:52:38.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:38.287+0000 7f71138c1640 1 -- 172.21.15.98:0/132431575 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f710c0008d0 con 0x7f710c0a9da0 2024-03-29T11:52:38.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:38.288+0000 7f7101ffb640 1 -- 172.21.15.98:0/132431575 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f7108050a70 con 0x7f710c0a9da0 2024-03-29T11:52:38.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:38.288+0000 7f7101ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:38.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:38.288+0000 7f7101ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:52:38.297 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:52:38.297 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:52:38.300 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:38.292+0000 7f71138c1640 10 librados: watch_flush enter 2024-03-29T11:52:38.300 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:38.293+0000 7f71138c1640 10 librados: watch_flush exit 2024-03-29T11:52:38.300 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:38.293+0000 7f71138c1640 20 client.4399.objecter shutdown clearing up homeless session... 2024-03-29T11:52:38.300 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:38.293+0000 7f71138c1640 10 client.4399.objecter successfully canceled tick 2024-03-29T11:52:38.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:38.294+0000 7f71138c1640 1 -- 172.21.15.98:0/132431575 >> v1:172.21.15.98:6816/4065186974 conn(0x7f70f0076630 legacy=0x7f70f0078af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:38.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:38.294+0000 7f71138c1640 10 monclient: shutdown 2024-03-29T11:52:38.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:38.294+0000 7f71138c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:52:38.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:38.294+0000 7f71138c1640 1 -- 172.21.15.98:0/132431575 >> v1:172.21.15.121:6789/0 conn(0x7f710c0a9da0 legacy=0x7f710c1d8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:38.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:38.294+0000 7f71138c1640 1 -- 172.21.15.98:0/132431575 shutdown_connections 2024-03-29T11:52:38.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:38.294+0000 7f71138c1640 1 -- 172.21.15.98:0/132431575 >> 172.21.15.98:0/132431575 conn(0x7f710c0b3c80 msgr2=0x7f710c0ac960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:52:38.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:38.296+0000 7f71138c1640 1 -- 172.21.15.98:0/132431575 shutdown_connections 2024-03-29T11:52:38.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:38.296+0000 7f71138c1640 1 -- 172.21.15.98:0/132431575 wait complete. 2024-03-29T11:52:38.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:38.296+0000 7f71138c1640 1 librados: shutdown 2024-03-29T11:52:38.320 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:52:45.127 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:52:45.283 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.274+0000 7f11ea2d9640 10 monclient: get_monmap_and_config 2024-03-29T11:52:45.283 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.274+0000 7f11ea2d9640 10 monclient: build_initial_monmap 2024-03-29T11:52:45.283 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.274+0000 7f11ea2d9640 10 monclient: monmap: 2024-03-29T11:52:45.283 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:52:45.283 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:45.283 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:52:45.275110+0000 2024-03-29T11:52:45.283 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:52:45.275110+0000 2024-03-29T11:52:45.283 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:52:45.283 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:45.283 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:52:45.283 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:52:45.283 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:52:45.284 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:45.284 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.276+0000 7f11ea2d9640 1 Processor -- start 2024-03-29T11:52:45.284 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.276+0000 7f11ea2d9640 1 -- start start 2024-03-29T11:52:45.284 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.276+0000 7f11ea2d9640 10 monclient: init 2024-03-29T11:52:45.284 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.277+0000 7f11ea2d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:52:45.285 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.277+0000 7f11ea2d9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:52:45.285 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.277+0000 7f11ea2d9640 10 monclient(hunting): picked mon.noname-b con 0x7f11e40b4100 addr v1:172.21.15.98:6790/0 2024-03-29T11:52:45.285 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.277+0000 7f11e3fff640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f11e40b4100 0x7f11e40a9850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:57158/0 (socket says 172.21.15.98:57158) 2024-03-29T11:52:45.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.278+0000 7f11ea2d9640 10 monclient(hunting): picked mon.noname-c con 0x7f11e413ee90 addr v1:172.21.15.121:6789/0 2024-03-29T11:52:45.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.278+0000 7f11ea2d9640 10 monclient(hunting): picked mon.noname-a con 0x7f11e40af900 addr v1:172.21.15.98:6789/0 2024-03-29T11:52:45.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.279+0000 7f11ea2d9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11e40afd10 con 0x7f11e40af900 2024-03-29T11:52:45.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.279+0000 7f11ea2d9640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11e40af6f0 con 0x7f11e413ee90 2024-03-29T11:52:45.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.279+0000 7f11ea2d9640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11e40aaf60 con 0x7f11e40b4100 2024-03-29T11:52:45.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.279+0000 7f11ea2d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:52:45.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.279+0000 7f11ea2d9640 10 monclient(hunting): authenticate will time out at 1072.364139s 2024-03-29T11:52:45.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.280+0000 7f11e8876640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f11e40af900 0x7f11e413b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:59856/0 (socket says 172.21.15.98:59856) 2024-03-29T11:52:45.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.280+0000 7f11e8876640 1 -- 172.21.15.98:0/3103657470 learned_addr learned my addr 172.21.15.98:0/3103657470 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:52:45.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.280+0000 7f11e2ffd640 1 -- 172.21.15.98:0/3103657470 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 22254360 0 0) 0x7f11e40afd10 con 0x7f11e40af900 2024-03-29T11:52:45.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.281+0000 7f11e2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:52:45.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.281+0000 7f11e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:45.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.281+0000 7f11e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:45.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.281+0000 7f11e2ffd640 10 monclient(hunting): my global_id is 4362 2024-03-29T11:52:45.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.281+0000 7f11e2ffd640 1 -- 172.21.15.98:0/3103657470 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11cc003520 con 0x7f11e40af900 2024-03-29T11:52:45.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.281+0000 7f11e2ffd640 1 -- 172.21.15.98:0/3103657470 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1860707642 0 0) 0x7f11cc003520 con 0x7f11e40af900 2024-03-29T11:52:45.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.281+0000 7f11e2ffd640 1 -- 172.21.15.98:0/3103657470 >> v1:172.21.15.98:6790/0 conn(0x7f11e40b4100 legacy=0x7f11e40a9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:52:45.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.281+0000 7f11e2ffd640 1 -- 172.21.15.98:0/3103657470 >> v1:172.21.15.121:6789/0 conn(0x7f11e413ee90 legacy=0x7f11e40adfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:52:45.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.281+0000 7f11e3fff640 1 --1- 172.21.15.98:0/3103657470 >> v1:172.21.15.98:6790/0 conn(0x7f11e40b4100 0x7f11e40a9850 :-1 s=CLOSED pgs=0 cs=0 l=0).handle_server_banner_and_identify state changed while learned_addr, mark_down or replacing must be happened just now 2024-03-29T11:52:45.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.281+0000 7f11e2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:52:45.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.281+0000 7f11e2ffd640 1 monclient: found mon.noname-a 2024-03-29T11:52:45.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.281+0000 7f11e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:52:45.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.281+0000 7f11e2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T11:52:45.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.281+0000 7f11e2ffd640 1 -- 172.21.15.98:0/3103657470 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e413e970 con 0x7f11e40af900 2024-03-29T11:52:45.291 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.281+0000 7f11e2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:52:45.291 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.281+0000 7f11e2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:52:45.291 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.281+0000 7f11e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:52:45.291 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.281+0000 7f11e2ffd640 1 -- 172.21.15.98:0/3103657470 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f11d8001690 con 0x7f11e40af900 2024-03-29T11:52:45.291 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.281+0000 7f11e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:45.291 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.281+0000 7f11e2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:52:45.291 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.281+0000 7f11e2ffd640 10 monclient: dump: 2024-03-29T11:52:45.291 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:52:45.291 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:45.291 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:45.291 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:45.291 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:45.291 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:45.291 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:52:45.291 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:52:45.292 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:52:45.292 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:45.292 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.282+0000 7f11ea2d9640 5 monclient: authenticate success, global_id 4362 2024-03-29T11:52:45.292 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.282+0000 7f11ea2d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:52:45.293 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.282+0000 7f11e2ffd640 1 -- 172.21.15.98:0/3103657470 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11d8001860 con 0x7f11e40af900 2024-03-29T11:52:45.293 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.282+0000 7f11e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:52:45.293 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.282+0000 7f11e2ffd640 1 -- 172.21.15.98:0/3103657470 <== mon.0 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f11d8001cb0 con 0x7f11e40af900 2024-03-29T11:52:45.293 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.282+0000 7f11e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:45.293 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.282+0000 7f11e2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:52:45.294 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.282+0000 7f11e2ffd640 10 monclient: dump: 2024-03-29T11:52:45.294 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:52:45.294 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:45.294 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:45.294 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:45.294 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:45.294 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:45.294 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:52:45.294 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:52:45.294 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:52:45.294 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:45.294 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.282+0000 7f11ea2d9640 10 monclient: get_monmap_and_config success 2024-03-29T11:52:45.294 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.282+0000 7f11ea2d9640 4 set_mon_vals no callback set 2024-03-29T11:52:45.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.283+0000 7f11ea2d9640 10 monclient: shutdown 2024-03-29T11:52:45.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.283+0000 7f11ea2d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:52:45.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.283+0000 7f11ea2d9640 1 -- 172.21.15.98:0/3103657470 >> v1:172.21.15.98:6789/0 conn(0x7f11e40af900 legacy=0x7f11e413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:45.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.283+0000 7f11ea2d9640 1 -- 172.21.15.98:0/3103657470 shutdown_connections 2024-03-29T11:52:45.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.283+0000 7f11ea2d9640 1 -- 172.21.15.98:0/3103657470 >> 172.21.15.98:0/3103657470 conn(0x7f11e40b35e0 msgr2=0x7f11e40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:52:45.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.284+0000 7f11ea2d9640 1 -- 172.21.15.98:0/3103657470 shutdown_connections 2024-03-29T11:52:45.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.284+0000 7f11ea2d9640 1 -- 172.21.15.98:0/3103657470 wait complete. 2024-03-29T11:52:45.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.284+0000 7f11ea2d9640 10 monclient: build_initial_monmap 2024-03-29T11:52:45.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.284+0000 7f11ea2d9640 10 monclient: monmap: 2024-03-29T11:52:45.295 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:52:45.295 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:52:45.295 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:52:45.295 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:52:45.295 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:52:45.296 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:45.296 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:52:45.296 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:52:45.296 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:52:45.296 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:45.296 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.285+0000 7f11ea2d9640 1 librados: starting msgr at 2024-03-29T11:52:45.296 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.285+0000 7f11ea2d9640 1 librados: starting objecter 2024-03-29T11:52:45.296 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.285+0000 7f11ea2d9640 1 Processor -- start 2024-03-29T11:52:45.296 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.286+0000 7f11ea2d9640 1 -- start start 2024-03-29T11:52:45.296 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.286+0000 7f11ea2d9640 1 librados: setting wanted keys 2024-03-29T11:52:45.296 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.286+0000 7f11ea2d9640 1 librados: calling monclient init 2024-03-29T11:52:45.296 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.286+0000 7f11ea2d9640 10 monclient: init 2024-03-29T11:52:45.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.287+0000 7f11ea2d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:52:45.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.287+0000 7f11ea2d9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:52:45.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.287+0000 7f11ea2d9640 10 monclient(hunting): picked mon.noname-b con 0x7f11e40af900 addr v1:172.21.15.121:6789/0 2024-03-29T11:52:45.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.287+0000 7f11ea2d9640 10 monclient(hunting): picked mon.noname-a con 0x7f11e40b4100 addr v1:172.21.15.98:6789/0 2024-03-29T11:52:45.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.287+0000 7f11ea2d9640 10 monclient(hunting): picked mon.noname-c con 0x7f11e413ee90 addr v1:172.21.15.98:6790/0 2024-03-29T11:52:45.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.287+0000 7f11ea2d9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11e41dbd80 con 0x7f11e40b4100 2024-03-29T11:52:45.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.287+0000 7f11ea2d9640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11e41dc210 con 0x7f11e40af900 2024-03-29T11:52:45.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.287+0000 7f11ea2d9640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11e41d18b0 con 0x7f11e413ee90 2024-03-29T11:52:45.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.287+0000 7f11ea2d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:52:45.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.287+0000 7f11ea2d9640 10 monclient(hunting): authenticate will time out at 1072.372674s 2024-03-29T11:52:45.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.290+0000 7f11e37fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f11e40b4100 0x7f11e406e7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:60014/0 (socket says 172.21.15.98:60014) 2024-03-29T11:52:45.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.290+0000 7f11e37fe640 1 -- 172.21.15.98:0/2868084054 learned_addr learned my addr 172.21.15.98:0/2868084054 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:52:45.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.290+0000 7f11e0ff9640 10 client.?.objecter ms_handle_connect 0x7f11e40b4100 2024-03-29T11:52:45.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.290+0000 7f11e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:52:45.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.290+0000 7f11e0ff9640 10 client.?.objecter ms_handle_connect 0x7f11e413ee90 2024-03-29T11:52:45.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.290+0000 7f11e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:52:45.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.290+0000 7f11e0ff9640 1 -- 172.21.15.98:0/2868084054 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 614400098 0 0) 0x7f11e41dbd80 con 0x7f11e40b4100 2024-03-29T11:52:45.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.290+0000 7f11e0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:52:45.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.290+0000 7f11e0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:45.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.290+0000 7f11e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:45.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.290+0000 7f11e0ff9640 10 monclient(hunting): my global_id is 4365 2024-03-29T11:52:45.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.290+0000 7f11e0ff9640 1 -- 172.21.15.98:0/2868084054 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11d0004eb0 con 0x7f11e40b4100 2024-03-29T11:52:45.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.291+0000 7f11e0ff9640 1 -- 172.21.15.98:0/2868084054 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 12064334 0 0) 0x7f11e41d18b0 con 0x7f11e413ee90 2024-03-29T11:52:45.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.291+0000 7f11e0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:52:45.300 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.291+0000 7f11e0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:45.300 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.291+0000 7f11e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:45.300 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.291+0000 7f11e0ff9640 10 monclient(hunting): my global_id is 4394 2024-03-29T11:52:45.300 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.291+0000 7f11e0ff9640 1 -- 172.21.15.98:0/2868084054 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11e41dbd80 con 0x7f11e413ee90 2024-03-29T11:52:45.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.291+0000 7f11e0ff9640 10 client.?.objecter ms_handle_connect 0x7f11e40af900 2024-03-29T11:52:45.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.291+0000 7f11e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:52:45.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.291+0000 7f11e0ff9640 1 -- 172.21.15.98:0/2868084054 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1548428187 0 0) 0x7f11e41dbd80 con 0x7f11e413ee90 2024-03-29T11:52:45.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.291+0000 7f11e0ff9640 1 -- 172.21.15.98:0/2868084054 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11e41d18b0 con 0x7f11e413ee90 2024-03-29T11:52:45.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.291+0000 7f11e0ff9640 1 -- 172.21.15.98:0/2868084054 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f11d8001b70 con 0x7f11e413ee90 2024-03-29T11:52:45.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.291+0000 7f11e0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:45.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.291+0000 7f11e0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:52:45.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.291+0000 7f11e0ff9640 10 monclient(hunting): dump: 2024-03-29T11:52:45.302 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:52:45.302 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:45.302 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:45.302 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:45.302 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:45.303 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:45.303 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:52:45.303 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:52:45.303 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:52:45.303 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:45.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.291+0000 7f11e0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:52:45.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.291+0000 7f11e0ff9640 1 -- 172.21.15.98:0/2868084054 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3040227214 0 0) 0x7f11d0004eb0 con 0x7f11e40b4100 2024-03-29T11:52:45.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.291+0000 7f11e0ff9640 1 -- 172.21.15.98:0/2868084054 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11e41dbd80 con 0x7f11e40b4100 2024-03-29T11:52:45.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.291+0000 7f11e0ff9640 1 -- 172.21.15.98:0/2868084054 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f11d4001750 con 0x7f11e40b4100 2024-03-29T11:52:45.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.291+0000 7f11e0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:45.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.291+0000 7f11e0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:52:45.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.291+0000 7f11e0ff9640 10 monclient(hunting): dump: 2024-03-29T11:52:45.303 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:52:45.303 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:45.303 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:45.303 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:45.304 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:45.304 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:45.304 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:52:45.304 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:52:45.304 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:52:45.304 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:45.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.291+0000 7f11e0ff9640 1 -- 172.21.15.98:0/2868084054 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1000043780 0 0) 0x7f11e41dc210 con 0x7f11e40af900 2024-03-29T11:52:45.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.291+0000 7f11e0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:52:45.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.291+0000 7f11e0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:45.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.291+0000 7f11e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:45.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.291+0000 7f11e0ff9640 10 monclient(hunting): my global_id is 4402 2024-03-29T11:52:45.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.291+0000 7f11e0ff9640 1 -- 172.21.15.98:0/2868084054 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11d0004eb0 con 0x7f11e40af900 2024-03-29T11:52:45.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.292+0000 7f11e0ff9640 1 -- 172.21.15.98:0/2868084054 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2080353473 0 0) 0x7f11d0004eb0 con 0x7f11e40af900 2024-03-29T11:52:45.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.293+0000 7f11e0ff9640 1 -- 172.21.15.98:0/2868084054 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11e41dc210 con 0x7f11e40af900 2024-03-29T11:52:45.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.293+0000 7f11e0ff9640 1 -- 172.21.15.98:0/2868084054 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f11dc001b10 con 0x7f11e40af900 2024-03-29T11:52:45.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.293+0000 7f11e0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:45.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.293+0000 7f11e0ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:52:45.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.293+0000 7f11e0ff9640 10 monclient(hunting): dump: 2024-03-29T11:52:45.305 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:52:45.305 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:45.305 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:45.305 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:45.305 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:45.305 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:45.305 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:52:45.305 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:52:45.305 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:52:45.305 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:45.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.293+0000 7f11e0ff9640 1 -- 172.21.15.98:0/2868084054 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3753563973 0 0) 0x7f11e41dc210 con 0x7f11e40af900 2024-03-29T11:52:45.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.293+0000 7f11e0ff9640 1 -- 172.21.15.98:0/2868084054 >> v1:172.21.15.98:6790/0 conn(0x7f11e413ee90 legacy=0x7f11e41e5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:45.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.293+0000 7f11e0ff9640 1 -- 172.21.15.98:0/2868084054 >> v1:172.21.15.98:6789/0 conn(0x7f11e40b4100 legacy=0x7f11e406e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:45.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.293+0000 7f11e0ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:52:45.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.293+0000 7f11e0ff9640 1 monclient: found mon.b 2024-03-29T11:52:45.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.293+0000 7f11e0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:52:45.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.293+0000 7f11e0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:52:45.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.293+0000 7f11e0ff9640 1 -- 172.21.15.98:0/2868084054 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e41d1a40 con 0x7f11e40af900 2024-03-29T11:52:45.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.293+0000 7f11e0ff9640 10 monclient: _finish_auth 0 2024-03-29T11:52:45.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.293+0000 7f11e0ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:52:45.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.293+0000 7f11e0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:52:45.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.293+0000 7f11ea2d9640 5 monclient: authenticate success, global_id 4402 2024-03-29T11:52:45.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.293+0000 7f11ea2d9640 10 monclient: _renew_subs 2024-03-29T11:52:45.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.293+0000 7f11ea2d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:52:45.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.294+0000 7f11ea2d9640 1 -- 172.21.15.98:0/2868084054 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11e406eeb0 con 0x7f11e40af900 2024-03-29T11:52:45.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.294+0000 7f11ea2d9640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:52:45.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.294+0000 7f11ea2d9640 10 monclient: _renew_subs 2024-03-29T11:52:45.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.294+0000 7f11ea2d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:52:45.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.294+0000 7f11ea2d9640 1 -- 172.21.15.98:0/2868084054 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f11e4141ba0 con 0x7f11e40af900 2024-03-29T11:52:45.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.294+0000 7f11ea2d9640 1 librados: init done 2024-03-29T11:52:45.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.294+0000 7f11e0ff9640 1 -- 172.21.15.98:0/2868084054 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11dc001c70 con 0x7f11e40af900 2024-03-29T11:52:45.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.294+0000 7f11e0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:52:45.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.294+0000 7f11e0ff9640 1 -- 172.21.15.98:0/2868084054 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f11dc004d00 con 0x7f11e40af900 2024-03-29T11:52:45.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.294+0000 7f11e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:45.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.294+0000 7f11e0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:52:45.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.294+0000 7f11e0ff9640 10 monclient: dump: 2024-03-29T11:52:45.308 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:52:45.308 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:45.309 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:45.309 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:45.309 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:45.309 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:45.309 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:52:45.309 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:52:45.309 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:52:45.309 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:45.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.294+0000 7f11e27fc640 4 set_mon_vals no callback set 2024-03-29T11:52:45.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.296+0000 7f11e0ff9640 1 -- 172.21.15.98:0/2868084054 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f11dc01acd0 con 0x7f11e40af900 2024-03-29T11:52:45.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.296+0000 7f11ea2d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:45.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.296+0000 7f11ea2d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:45.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.296+0000 7f11ea2d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:52:45.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.296+0000 7f11ea2d9640 1 -- 172.21.15.98:0/2868084054 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11c4005080 con 0x7f11e40af900 2024-03-29T11:52:45.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.297+0000 7f11e37fe640 10 monclient: get_auth_request con 0x7f11d006e1b0 auth_method 0 2024-03-29T11:52:45.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.297+0000 7f11e0ff9640 1 -- 172.21.15.98:0/2868084054 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f11dc084cc0 con 0x7f11e40af900 2024-03-29T11:52:45.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.297+0000 7f11e0ff9640 10 client.4402.objecter ms_dispatch 0x7f11e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:52:45.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.297+0000 7f11e0ff9640 3 client.4402.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:52:45.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.297+0000 7f11e0ff9640 3 client.4402.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:52:45.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.297+0000 7f11e0ff9640 20 client.4402.objecter dump_active .. 0 homeless 2024-03-29T11:52:45.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.297+0000 7f11e0ff9640 10 client.4402.objecter ms_handle_connect 0x7f11d006e1b0 2024-03-29T11:52:45.314 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.307+0000 7f11e0ff9640 1 -- 172.21.15.98:0/2868084054 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f11dc0505a0 con 0x7f11e40af900 2024-03-29T11:52:45.315 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.307+0000 7f11e0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:45.315 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.307+0000 7f11e0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:52:45.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.650+0000 7f11ea2d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:45.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.650+0000 7f11ea2d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:45.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.650+0000 7f11ea2d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:52:45.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.650+0000 7f11ea2d9640 1 -- 172.21.15.98:0/2868084054 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11c4005330 con 0x7f11e40af900 2024-03-29T11:52:45.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.651+0000 7f11e0ff9640 1 -- 172.21.15.98:0/2868084054 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f11dc0524d0 con 0x7f11e40af900 2024-03-29T11:52:45.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.651+0000 7f11e0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:45.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.651+0000 7f11e0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:52:45.659 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:52:45.659 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:52:45.663 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.654+0000 7f11ea2d9640 10 librados: watch_flush enter 2024-03-29T11:52:45.663 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.655+0000 7f11ea2d9640 10 librados: watch_flush exit 2024-03-29T11:52:45.663 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.655+0000 7f11ea2d9640 20 client.4402.objecter shutdown clearing up homeless session... 2024-03-29T11:52:45.663 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.655+0000 7f11ea2d9640 10 client.4402.objecter successfully canceled tick 2024-03-29T11:52:45.663 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.655+0000 7f11ea2d9640 1 -- 172.21.15.98:0/2868084054 >> v1:172.21.15.98:6816/4065186974 conn(0x7f11d006e1b0 legacy=0x7f11d0070670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:45.664 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.655+0000 7f11ea2d9640 10 monclient: shutdown 2024-03-29T11:52:45.664 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.655+0000 7f11ea2d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:52:45.664 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.655+0000 7f11ea2d9640 1 -- 172.21.15.98:0/2868084054 >> v1:172.21.15.121:6789/0 conn(0x7f11e40af900 legacy=0x7f11e41db4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:45.664 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.655+0000 7f11ea2d9640 1 -- 172.21.15.98:0/2868084054 shutdown_connections 2024-03-29T11:52:45.664 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.655+0000 7f11ea2d9640 1 -- 172.21.15.98:0/2868084054 >> 172.21.15.98:0/2868084054 conn(0x7f11e40b35e0 msgr2=0x7f11e40acb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:52:45.664 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.656+0000 7f11ea2d9640 1 -- 172.21.15.98:0/2868084054 shutdown_connections 2024-03-29T11:52:45.664 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.656+0000 7f11ea2d9640 1 -- 172.21.15.98:0/2868084054 wait complete. 2024-03-29T11:52:45.664 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:45.656+0000 7f11ea2d9640 1 librados: shutdown 2024-03-29T11:52:45.679 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:52:52.486 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:52:52.636 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.627+0000 7fa0a97c4640 10 monclient: get_monmap_and_config 2024-03-29T11:52:52.637 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.627+0000 7fa0a97c4640 10 monclient: build_initial_monmap 2024-03-29T11:52:52.638 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.627+0000 7fa0a97c4640 10 monclient: monmap: 2024-03-29T11:52:52.638 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:52:52.638 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:52.638 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:52:52.629480+0000 2024-03-29T11:52:52.638 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:52:52.629480+0000 2024-03-29T11:52:52.638 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:52:52.638 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:52.638 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:52:52.638 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:52:52.638 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:52:52.638 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:52.638 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.628+0000 7fa0a97c4640 1 Processor -- start 2024-03-29T11:52:52.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.628+0000 7fa0a97c4640 1 -- start start 2024-03-29T11:52:52.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.628+0000 7fa0a97c4640 10 monclient: init 2024-03-29T11:52:52.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.629+0000 7fa0a97c4640 10 monclient: _reopen_session rank -1 2024-03-29T11:52:52.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.629+0000 7fa0a97c4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:52:52.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.629+0000 7fa0a97c4640 10 monclient(hunting): picked mon.noname-c con 0x7fa0a40a9930 addr v1:172.21.15.121:6789/0 2024-03-29T11:52:52.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.629+0000 7fa0a97c4640 10 monclient(hunting): picked mon.noname-b con 0x7fa0a4149dc0 addr v1:172.21.15.98:6790/0 2024-03-29T11:52:52.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.629+0000 7fa0a97c4640 10 monclient(hunting): picked mon.noname-a con 0x7fa0a414ff40 addr v1:172.21.15.98:6789/0 2024-03-29T11:52:52.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.629+0000 7fa0a97c4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0a40aae60 con 0x7fa0a414ff40 2024-03-29T11:52:52.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.629+0000 7fa0a97c4640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0a40b4310 con 0x7fa0a40a9930 2024-03-29T11:52:52.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.629+0000 7fa0a97c4640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0a40afdd0 con 0x7fa0a4149dc0 2024-03-29T11:52:52.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.629+0000 7fa0a97c4640 10 monclient(hunting): _renew_subs 2024-03-29T11:52:52.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.629+0000 7fa0a97c4640 10 monclient(hunting): authenticate will time out at 1079.715742s 2024-03-29T11:52:52.640 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.629+0000 7fa0a2ffd640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fa0a40a9930 0x7fa0a4143a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:44266/0 (socket says 172.21.15.98:44266) 2024-03-29T11:52:52.640 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.629+0000 7fa0a2ffd640 1 -- 172.21.15.98:0/1078308528 learned_addr learned my addr 172.21.15.98:0/1078308528 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:52:52.640 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.629+0000 7fa0a27fc640 1 --1- 172.21.15.98:0/1078308528 >> v1:172.21.15.98:6790/0 conn(0x7fa0a4149dc0 0x7fa0a414a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:53294/0 (socket says 172.21.15.98:53294) 2024-03-29T11:52:52.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.630+0000 7fa0a1ffb640 1 -- 172.21.15.98:0/1078308528 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 474032091 0 0) 0x7fa0a40b4310 con 0x7fa0a40a9930 2024-03-29T11:52:52.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.630+0000 7fa0a1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:52:52.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.630+0000 7fa0a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:52.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.630+0000 7fa0a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:52.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.630+0000 7fa0a1ffb640 10 monclient(hunting): my global_id is 4405 2024-03-29T11:52:52.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.630+0000 7fa0a1ffb640 1 -- 172.21.15.98:0/1078308528 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa090003520 con 0x7fa0a40a9930 2024-03-29T11:52:52.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.630+0000 7fa0a1ffb640 1 -- 172.21.15.98:0/1078308528 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3236822430 0 0) 0x7fa0a40afdd0 con 0x7fa0a4149dc0 2024-03-29T11:52:52.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.630+0000 7fa0a1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:52:52.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.630+0000 7fa0a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:52.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.630+0000 7fa0a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:52.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.630+0000 7fa0a1ffb640 10 monclient(hunting): my global_id is 4397 2024-03-29T11:52:52.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.630+0000 7fa0a1ffb640 1 -- 172.21.15.98:0/1078308528 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0a40b4310 con 0x7fa0a4149dc0 2024-03-29T11:52:52.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.630+0000 7fa0a1ffb640 1 -- 172.21.15.98:0/1078308528 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3157797703 0 0) 0x7fa0a40aae60 con 0x7fa0a414ff40 2024-03-29T11:52:52.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.630+0000 7fa0a1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:52:52.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.630+0000 7fa0a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:52.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.630+0000 7fa0a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:52.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.630+0000 7fa0a1ffb640 10 monclient(hunting): my global_id is 4368 2024-03-29T11:52:52.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.630+0000 7fa0a1ffb640 1 -- 172.21.15.98:0/1078308528 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0a40afdd0 con 0x7fa0a414ff40 2024-03-29T11:52:52.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.631+0000 7fa0a1ffb640 1 -- 172.21.15.98:0/1078308528 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3489102725 0 0) 0x7fa090003520 con 0x7fa0a40a9930 2024-03-29T11:52:52.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.631+0000 7fa0a1ffb640 1 -- 172.21.15.98:0/1078308528 >> v1:172.21.15.98:6790/0 conn(0x7fa0a4149dc0 legacy=0x7fa0a414a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:52.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.631+0000 7fa0a1ffb640 1 -- 172.21.15.98:0/1078308528 >> v1:172.21.15.98:6789/0 conn(0x7fa0a414ff40 legacy=0x7fa0a4152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:52.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.631+0000 7fa0a1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:52:52.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.631+0000 7fa0a1ffb640 1 monclient: found mon.noname-c 2024-03-29T11:52:52.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.631+0000 7fa0a1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:52:52.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.631+0000 7fa0a1ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:52:52.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.631+0000 7fa0a1ffb640 1 -- 172.21.15.98:0/1078308528 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a414a950 con 0x7fa0a40a9930 2024-03-29T11:52:52.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.631+0000 7fa0a1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:52:52.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.631+0000 7fa0a1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:52:52.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.631+0000 7fa0a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:52:52.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.631+0000 7fa0a1ffb640 1 -- 172.21.15.98:0/1078308528 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fa08c001690 con 0x7fa0a40a9930 2024-03-29T11:52:52.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.631+0000 7fa0a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:52.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.631+0000 7fa0a1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:52:52.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.631+0000 7fa0a1ffb640 10 monclient: dump: 2024-03-29T11:52:52.651 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:52:52.651 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:52.651 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:52.651 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:52.652 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:52.652 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:52.652 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:52:52.652 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:52:52.652 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:52:52.652 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:52.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.631+0000 7fa0a1ffb640 1 -- 172.21.15.98:0/1078308528 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa08c0018b0 con 0x7fa0a40a9930 2024-03-29T11:52:52.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.631+0000 7fa0a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:52:52.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.631+0000 7fa0a1ffb640 1 -- 172.21.15.98:0/1078308528 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fa08c001c40 con 0x7fa0a40a9930 2024-03-29T11:52:52.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.631+0000 7fa0a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:52.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.631+0000 7fa0a1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:52:52.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.631+0000 7fa0a1ffb640 10 monclient: dump: 2024-03-29T11:52:52.652 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:52:52.652 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:52.653 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:52.653 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:52.653 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:52.653 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:52.653 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:52:52.653 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:52:52.653 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:52:52.653 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:52.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.631+0000 7fa0a97c4640 5 monclient: authenticate success, global_id 4405 2024-03-29T11:52:52.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.631+0000 7fa0a97c4640 10 monclient: get_monmap_and_config success 2024-03-29T11:52:52.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.631+0000 7fa0a97c4640 4 set_mon_vals no callback set 2024-03-29T11:52:52.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.633+0000 7fa0a97c4640 10 monclient: shutdown 2024-03-29T11:52:52.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.633+0000 7fa0a97c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:52:52.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.633+0000 7fa0a97c4640 1 -- 172.21.15.98:0/1078308528 >> v1:172.21.15.121:6789/0 conn(0x7fa0a40a9930 legacy=0x7fa0a4143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:52.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.633+0000 7fa0a97c4640 1 -- 172.21.15.98:0/1078308528 shutdown_connections 2024-03-29T11:52:52.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.633+0000 7fa0a97c4640 1 -- 172.21.15.98:0/1078308528 >> 172.21.15.98:0/1078308528 conn(0x7fa0a40b35e0 msgr2=0x7fa0a40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:52:52.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.633+0000 7fa0a97c4640 1 -- 172.21.15.98:0/1078308528 shutdown_connections 2024-03-29T11:52:52.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.633+0000 7fa0a97c4640 1 -- 172.21.15.98:0/1078308528 wait complete. 2024-03-29T11:52:52.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.634+0000 7fa0a97c4640 10 monclient: build_initial_monmap 2024-03-29T11:52:52.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.634+0000 7fa0a97c4640 10 monclient: monmap: 2024-03-29T11:52:52.654 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:52:52.654 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:52:52.654 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:52:52.654 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:52:52.654 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:52:52.654 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:52.654 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:52:52.654 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:52:52.654 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:52:52.654 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:52.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.634+0000 7fa0a97c4640 1 librados: starting msgr at 2024-03-29T11:52:52.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.634+0000 7fa0a97c4640 1 librados: starting objecter 2024-03-29T11:52:52.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.635+0000 7fa0a97c4640 1 Processor -- start 2024-03-29T11:52:52.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.635+0000 7fa0a97c4640 1 -- start start 2024-03-29T11:52:52.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.635+0000 7fa0a97c4640 1 librados: setting wanted keys 2024-03-29T11:52:52.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.635+0000 7fa0a97c4640 1 librados: calling monclient init 2024-03-29T11:52:52.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.635+0000 7fa0a97c4640 10 monclient: init 2024-03-29T11:52:52.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.636+0000 7fa0a97c4640 10 monclient: _reopen_session rank -1 2024-03-29T11:52:52.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.636+0000 7fa0a97c4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:52:52.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.636+0000 7fa0a97c4640 10 monclient(hunting): picked mon.noname-c con 0x7fa0a40a9930 addr v1:172.21.15.98:6790/0 2024-03-29T11:52:52.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.636+0000 7fa0a97c4640 10 monclient(hunting): picked mon.noname-a con 0x7fa0a4149dc0 addr v1:172.21.15.98:6789/0 2024-03-29T11:52:52.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.636+0000 7fa0a97c4640 10 monclient(hunting): picked mon.noname-b con 0x7fa0a414ff40 addr v1:172.21.15.121:6789/0 2024-03-29T11:52:52.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.636+0000 7fa0a97c4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0a41db880 con 0x7fa0a4149dc0 2024-03-29T11:52:52.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.636+0000 7fa0a97c4640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0a4147e90 con 0x7fa0a414ff40 2024-03-29T11:52:52.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.636+0000 7fa0a97c4640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0a4148020 con 0x7fa0a40a9930 2024-03-29T11:52:52.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.636+0000 7fa0a97c4640 10 monclient(hunting): _renew_subs 2024-03-29T11:52:52.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.636+0000 7fa0a97c4640 10 monclient(hunting): authenticate will time out at 1079.722497s 2024-03-29T11:52:52.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.636+0000 7fa0a27fc640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa0a4149dc0 0x7fa0a413f770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56144/0 (socket says 172.21.15.98:56144) 2024-03-29T11:52:52.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.636+0000 7fa0a27fc640 1 -- 172.21.15.98:0/2558641029 learned_addr learned my addr 172.21.15.98:0/2558641029 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:52:52.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.636+0000 7fa09b7fe640 10 client.?.objecter ms_handle_connect 0x7fa0a40a9930 2024-03-29T11:52:52.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.636+0000 7fa09b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:52:52.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.636+0000 7fa09b7fe640 10 client.?.objecter ms_handle_connect 0x7fa0a4149dc0 2024-03-29T11:52:52.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.636+0000 7fa09b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:52:52.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.636+0000 7fa09b7fe640 10 client.?.objecter ms_handle_connect 0x7fa0a414ff40 2024-03-29T11:52:52.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.637+0000 7fa09b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:52:52.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.637+0000 7fa09b7fe640 1 -- 172.21.15.98:0/2558641029 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4215806732 0 0) 0x7fa0a4148020 con 0x7fa0a40a9930 2024-03-29T11:52:52.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.637+0000 7fa09b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:52:52.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.637+0000 7fa09b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:52.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.637+0000 7fa09b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:52.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.637+0000 7fa09b7fe640 10 monclient(hunting): my global_id is 4400 2024-03-29T11:52:52.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.637+0000 7fa09b7fe640 1 -- 172.21.15.98:0/2558641029 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa084004eb0 con 0x7fa0a40a9930 2024-03-29T11:52:52.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.637+0000 7fa09b7fe640 1 -- 172.21.15.98:0/2558641029 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2898051191 0 0) 0x7fa0a4147e90 con 0x7fa0a414ff40 2024-03-29T11:52:52.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.637+0000 7fa09b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:52:52.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.637+0000 7fa09b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:52.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.637+0000 7fa09b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:52.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.637+0000 7fa09b7fe640 10 monclient(hunting): my global_id is 4408 2024-03-29T11:52:52.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.637+0000 7fa09b7fe640 1 -- 172.21.15.98:0/2558641029 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0a4148020 con 0x7fa0a414ff40 2024-03-29T11:52:52.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.637+0000 7fa09b7fe640 1 -- 172.21.15.98:0/2558641029 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1998791200 0 0) 0x7fa0a41db880 con 0x7fa0a4149dc0 2024-03-29T11:52:52.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.637+0000 7fa09b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:52:52.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.637+0000 7fa09b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:52:52.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.637+0000 7fa09b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:52:52.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.637+0000 7fa09b7fe640 10 monclient(hunting): my global_id is 4371 2024-03-29T11:52:52.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.637+0000 7fa09b7fe640 1 -- 172.21.15.98:0/2558641029 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0a4147e90 con 0x7fa0a4149dc0 2024-03-29T11:52:52.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.637+0000 7fa09b7fe640 1 -- 172.21.15.98:0/2558641029 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2920131647 0 0) 0x7fa084004eb0 con 0x7fa0a40a9930 2024-03-29T11:52:52.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.637+0000 7fa09b7fe640 1 -- 172.21.15.98:0/2558641029 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0a41db880 con 0x7fa0a40a9930 2024-03-29T11:52:52.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.637+0000 7fa09b7fe640 1 -- 172.21.15.98:0/2558641029 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fa08c0015d0 con 0x7fa0a40a9930 2024-03-29T11:52:52.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.637+0000 7fa09b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:52.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.637+0000 7fa09b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:52:52.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.637+0000 7fa09b7fe640 10 monclient(hunting): dump: 2024-03-29T11:52:52.658 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:52:52.658 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:52.658 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:52.658 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:52.658 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:52.658 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:52.658 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:52:52.658 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:52:52.658 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:52:52.658 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:52.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.638+0000 7fa09b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:52:52.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.638+0000 7fa09b7fe640 1 -- 172.21.15.98:0/2558641029 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 169984198 0 0) 0x7fa0a4148020 con 0x7fa0a414ff40 2024-03-29T11:52:52.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.638+0000 7fa09b7fe640 1 -- 172.21.15.98:0/2558641029 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa084004eb0 con 0x7fa0a414ff40 2024-03-29T11:52:52.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.638+0000 7fa09b7fe640 1 -- 172.21.15.98:0/2558641029 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fa09c001b50 con 0x7fa0a414ff40 2024-03-29T11:52:52.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.638+0000 7fa09b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:52.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.638+0000 7fa09b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:52:52.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.638+0000 7fa09b7fe640 10 monclient(hunting): dump: 2024-03-29T11:52:52.659 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:52:52.659 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:52.659 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:52.659 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:52.659 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:52.659 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:52.659 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:52:52.660 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:52:52.660 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:52:52.660 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:52.660 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.638+0000 7fa09b7fe640 1 -- 172.21.15.98:0/2558641029 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 278256865 0 0) 0x7fa0a4147e90 con 0x7fa0a4149dc0 2024-03-29T11:52:52.660 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.638+0000 7fa09b7fe640 1 -- 172.21.15.98:0/2558641029 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0a4148020 con 0x7fa0a4149dc0 2024-03-29T11:52:52.660 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.638+0000 7fa09b7fe640 1 -- 172.21.15.98:0/2558641029 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fa0940018b0 con 0x7fa0a4149dc0 2024-03-29T11:52:52.660 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.638+0000 7fa09b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:52.660 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.638+0000 7fa09b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:52:52.660 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.638+0000 7fa09b7fe640 10 monclient(hunting): dump: 2024-03-29T11:52:52.660 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:52:52.660 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:52.660 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:52.660 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:52.660 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:52.661 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:52.661 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:52:52.661 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:52:52.661 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:52:52.661 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:52.661 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.638+0000 7fa09b7fe640 1 -- 172.21.15.98:0/2558641029 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4195298083 0 0) 0x7fa084004eb0 con 0x7fa0a414ff40 2024-03-29T11:52:52.661 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.638+0000 7fa09b7fe640 1 -- 172.21.15.98:0/2558641029 >> v1:172.21.15.98:6790/0 conn(0x7fa0a40a9930 legacy=0x7fa0a41dbd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:52.661 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.638+0000 7fa09b7fe640 1 -- 172.21.15.98:0/2558641029 >> v1:172.21.15.98:6789/0 conn(0x7fa0a4149dc0 legacy=0x7fa0a413f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:52.661 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.638+0000 7fa09b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:52:52.661 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.638+0000 7fa09b7fe640 1 monclient: found mon.b 2024-03-29T11:52:52.661 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.638+0000 7fa09b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:52:52.661 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.638+0000 7fa09b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:52:52.661 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.638+0000 7fa09b7fe640 1 -- 172.21.15.98:0/2558641029 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a41dc490 con 0x7fa0a414ff40 2024-03-29T11:52:52.661 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.638+0000 7fa09b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:52:52.661 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.638+0000 7fa09b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:52:52.662 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.638+0000 7fa09b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:52:52.662 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.638+0000 7fa0a97c4640 5 monclient: authenticate success, global_id 4408 2024-03-29T11:52:52.662 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.638+0000 7fa0a97c4640 10 monclient: _renew_subs 2024-03-29T11:52:52.662 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.638+0000 7fa0a97c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:52:52.662 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.638+0000 7fa0a97c4640 1 -- 172.21.15.98:0/2558641029 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0a41dc6a0 con 0x7fa0a414ff40 2024-03-29T11:52:52.662 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.638+0000 7fa0a97c4640 10 client.4408.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:52:52.662 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.638+0000 7fa0a97c4640 10 monclient: _renew_subs 2024-03-29T11:52:52.662 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.638+0000 7fa0a97c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:52:52.662 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.638+0000 7fa0a97c4640 1 -- 172.21.15.98:0/2558641029 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0a41400c0 con 0x7fa0a414ff40 2024-03-29T11:52:52.662 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.638+0000 7fa0a97c4640 1 librados: init done 2024-03-29T11:52:52.662 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.638+0000 7fa09b7fe640 1 -- 172.21.15.98:0/2558641029 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa09c002020 con 0x7fa0a414ff40 2024-03-29T11:52:52.662 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.638+0000 7fa09b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:52:52.662 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.638+0000 7fa09b7fe640 1 -- 172.21.15.98:0/2558641029 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fa09c003b10 con 0x7fa0a414ff40 2024-03-29T11:52:52.663 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.638+0000 7fa09b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:52:52.663 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.638+0000 7fa09b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:52:52.663 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.638+0000 7fa09b7fe640 10 monclient: dump: 2024-03-29T11:52:52.663 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:52:52.663 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:52:52.663 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:52.663 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:52:52.663 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:52:52.663 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:52:52.663 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:52:52.663 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:52:52.663 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:52:52.663 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:52:52.663 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.639+0000 7fa0a17fa640 4 set_mon_vals no callback set 2024-03-29T11:52:52.664 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.640+0000 7fa0a97c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:52.664 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.640+0000 7fa0a97c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:52.664 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.640+0000 7fa0a97c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:52:52.664 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.640+0000 7fa0a97c4640 1 -- 172.21.15.98:0/2558641029 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa088005080 con 0x7fa0a414ff40 2024-03-29T11:52:52.665 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.642+0000 7fa09b7fe640 1 -- 172.21.15.98:0/2558641029 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7fa09c019ae0 con 0x7fa0a414ff40 2024-03-29T11:52:52.665 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.643+0000 7fa0a2ffd640 10 monclient: get_auth_request con 0x7fa08406e150 auth_method 0 2024-03-29T11:52:52.665 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.643+0000 7fa09b7fe640 10 client.4408.objecter ms_handle_connect 0x7fa08406e150 2024-03-29T11:52:52.665 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.643+0000 7fa09b7fe640 1 -- 172.21.15.98:0/2558641029 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7fa09c0839a0 con 0x7fa0a414ff40 2024-03-29T11:52:52.665 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.643+0000 7fa09b7fe640 10 client.4408.objecter ms_dispatch 0x7fa0a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:52:52.665 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.643+0000 7fa09b7fe640 3 client.4408.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:52:52.665 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.643+0000 7fa09b7fe640 3 client.4408.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:52:52.665 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.643+0000 7fa09b7fe640 20 client.4408.objecter dump_active .. 0 homeless 2024-03-29T11:52:52.666 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.650+0000 7fa09b7fe640 1 -- 172.21.15.98:0/2558641029 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa09c04f280 con 0x7fa0a414ff40 2024-03-29T11:52:52.666 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.650+0000 7fa09b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:52:52.666 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:52.650+0000 7fa09b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:52:53.030 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:53.022+0000 7fa0a97c4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:53.031 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:53.022+0000 7fa0a97c4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:53.031 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:53.022+0000 7fa0a97c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:52:53.031 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:53.022+0000 7fa0a97c4640 1 -- 172.21.15.98:0/2558641029 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa088005330 con 0x7fa0a414ff40 2024-03-29T11:52:53.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:53.024+0000 7fa09b7fe640 1 -- 172.21.15.98:0/2558641029 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7fa09c0511b0 con 0x7fa0a414ff40 2024-03-29T11:52:53.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:53.024+0000 7fa09b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:52:53.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:53.024+0000 7fa09b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:52:53.033 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:52:53.033 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:52:53.036 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:53.027+0000 7fa0a97c4640 10 librados: watch_flush enter 2024-03-29T11:52:53.036 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:53.027+0000 7fa0a97c4640 10 librados: watch_flush exit 2024-03-29T11:52:53.036 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:53.027+0000 7fa0a97c4640 20 client.4408.objecter shutdown clearing up homeless session... 2024-03-29T11:52:53.036 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:53.027+0000 7fa0a97c4640 10 client.4408.objecter successfully canceled tick 2024-03-29T11:52:53.036 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:53.028+0000 7fa0a97c4640 1 -- 172.21.15.98:0/2558641029 >> v1:172.21.15.98:6816/4065186974 conn(0x7fa08406e150 legacy=0x7fa084070610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:53.036 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:53.028+0000 7fa0a97c4640 10 monclient: shutdown 2024-03-29T11:52:53.036 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:53.028+0000 7fa0a97c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:52:53.036 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:53.028+0000 7fa0a97c4640 1 -- 172.21.15.98:0/2558641029 >> v1:172.21.15.121:6789/0 conn(0x7fa0a414ff40 legacy=0x7fa0a41e6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:52:53.036 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:53.028+0000 7fa0a97c4640 1 -- 172.21.15.98:0/2558641029 shutdown_connections 2024-03-29T11:52:53.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:53.028+0000 7fa0a97c4640 1 -- 172.21.15.98:0/2558641029 >> 172.21.15.98:0/2558641029 conn(0x7fa0a40b35e0 msgr2=0x7fa0a40adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:52:53.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:53.028+0000 7fa0a97c4640 1 -- 172.21.15.98:0/2558641029 shutdown_connections 2024-03-29T11:52:53.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:53.028+0000 7fa0a97c4640 1 -- 172.21.15.98:0/2558641029 wait complete. 2024-03-29T11:52:53.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:53.028+0000 7fa0a97c4640 1 librados: shutdown 2024-03-29T11:52:53.052 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:52:59.858 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:53:00.000 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.992+0000 7f4b47ef1640 10 monclient: get_monmap_and_config 2024-03-29T11:53:00.000 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.992+0000 7f4b47ef1640 10 monclient: build_initial_monmap 2024-03-29T11:53:00.000 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.992+0000 7f4b47ef1640 10 monclient: monmap: 2024-03-29T11:53:00.000 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:53:00.000 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:00.000 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:52:59.993118+0000 2024-03-29T11:53:00.001 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:52:59.993118+0000 2024-03-29T11:53:00.001 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:53:00.001 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:00.001 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:53:00.001 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:53:00.001 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:53:00.001 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:00.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.993+0000 7f4b47ef1640 1 Processor -- start 2024-03-29T11:53:00.002 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.993+0000 7f4b47ef1640 1 -- start start 2024-03-29T11:53:00.002 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.993+0000 7f4b47ef1640 10 monclient: init 2024-03-29T11:53:00.002 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.994+0000 7f4b47ef1640 10 monclient: _reopen_session rank -1 2024-03-29T11:53:00.002 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.994+0000 7f4b47ef1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:53:00.002 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.994+0000 7f4b47ef1640 10 monclient(hunting): picked mon.noname-b con 0x7f4b401457f0 addr v1:172.21.15.98:6790/0 2024-03-29T11:53:00.002 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.994+0000 7f4b45c8d640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f4b401457f0 0x7f4b4013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:35106/0 (socket says 172.21.15.98:35106) 2024-03-29T11:53:00.004 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.996+0000 7f4b47ef1640 10 monclient(hunting): picked mon.noname-a con 0x7f4b4013f650 addr v1:172.21.15.98:6789/0 2024-03-29T11:53:00.004 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.996+0000 7f4b47ef1640 10 monclient(hunting): picked mon.noname-c con 0x7f4b400a9d70 addr v1:172.21.15.121:6789/0 2024-03-29T11:53:00.005 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.996+0000 7f4b47ef1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b400aaef0 con 0x7f4b4013f650 2024-03-29T11:53:00.005 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.996+0000 7f4b47ef1640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b40145bd0 con 0x7f4b400a9d70 2024-03-29T11:53:00.005 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.996+0000 7f4b47ef1640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b40145eb0 con 0x7f4b401457f0 2024-03-29T11:53:00.005 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.996+0000 7f4b47ef1640 10 monclient(hunting): _renew_subs 2024-03-29T11:53:00.005 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.996+0000 7f4b47ef1640 10 monclient(hunting): authenticate will time out at 1087.081931s 2024-03-29T11:53:00.005 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.997+0000 7f4b45c8d640 1 -- 172.21.15.98:0/625840476 learned_addr learned my addr 172.21.15.98:0/625840476 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:53:00.005 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.998+0000 7f4b44c8b640 1 -- 172.21.15.98:0/625840476 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3055105136 0 0) 0x7f4b400aaef0 con 0x7f4b4013f650 2024-03-29T11:53:00.005 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.998+0000 7f4b44c8b640 10 monclient(hunting): _negotiate 2024-03-29T11:53:00.005 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.998+0000 7f4b44c8b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:00.006 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.998+0000 7f4b44c8b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:00.006 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.998+0000 7f4b44c8b640 10 monclient(hunting): my global_id is 4374 2024-03-29T11:53:00.006 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.998+0000 7f4b44c8b640 1 -- 172.21.15.98:0/625840476 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b2c003520 con 0x7f4b4013f650 2024-03-29T11:53:00.006 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.998+0000 7f4b44c8b640 1 -- 172.21.15.98:0/625840476 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2651291780 0 0) 0x7f4b40145eb0 con 0x7f4b401457f0 2024-03-29T11:53:00.006 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.998+0000 7f4b44c8b640 10 monclient(hunting): _negotiate 2024-03-29T11:53:00.006 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.998+0000 7f4b44c8b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:00.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.998+0000 7f4b44c8b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:00.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.998+0000 7f4b44c8b640 10 monclient(hunting): my global_id is 4403 2024-03-29T11:53:00.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.998+0000 7f4b44c8b640 1 -- 172.21.15.98:0/625840476 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b400aaef0 con 0x7f4b401457f0 2024-03-29T11:53:00.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.999+0000 7f4b44c8b640 1 -- 172.21.15.98:0/625840476 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1146849834 0 0) 0x7f4b2c003520 con 0x7f4b4013f650 2024-03-29T11:53:00.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.999+0000 7f4b44c8b640 1 -- 172.21.15.98:0/625840476 >> v1:172.21.15.98:6790/0 conn(0x7f4b401457f0 legacy=0x7f4b4013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:00.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.999+0000 7f4b44c8b640 1 -- 172.21.15.98:0/625840476 >> v1:172.21.15.121:6789/0 conn(0x7f4b400a9d70 legacy=0x7f4b4014d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:53:00.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.999+0000 7f4b44c8b640 10 monclient: _finish_hunting 0 2024-03-29T11:53:00.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.999+0000 7f4b44c8b640 1 monclient: found mon.noname-a 2024-03-29T11:53:00.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.999+0000 7f4b44c8b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:53:00.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.999+0000 7f4b44c8b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T11:53:00.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.999+0000 7f4b44c8b640 1 -- 172.21.15.98:0/625840476 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b40146340 con 0x7f4b4013f650 2024-03-29T11:53:00.008 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.999+0000 7f4b44c8b640 10 monclient: _finish_auth 0 2024-03-29T11:53:00.008 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.999+0000 7f4b44c8b640 10 monclient: _check_auth_tickets 2024-03-29T11:53:00.008 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.999+0000 7f4b44c8b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:53:00.008 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.999+0000 7f4b44c8b640 1 -- 172.21.15.98:0/625840476 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f4b300015c0 con 0x7f4b4013f650 2024-03-29T11:53:00.008 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.999+0000 7f4b44c8b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:00.008 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.999+0000 7f4b44c8b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:53:00.008 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.999+0000 7f4b44c8b640 10 monclient: dump: 2024-03-29T11:53:00.008 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:53:00.008 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:00.008 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:00.008 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:00.008 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:00.008 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:00.008 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:53:00.009 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:53:00.009 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:53:00.009 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:00.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.999+0000 7f4b47ef1640 5 monclient: authenticate success, global_id 4374 2024-03-29T11:53:00.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:52:59.999+0000 7f4b47ef1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:53:00.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.001+0000 7f4b44c8b640 1 -- 172.21.15.98:0/625840476 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b30001720 con 0x7f4b4013f650 2024-03-29T11:53:00.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.001+0000 7f4b44c8b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:53:00.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.001+0000 7f4b44c8b640 1 -- 172.21.15.98:0/625840476 <== mon.0 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f4b30001b70 con 0x7f4b4013f650 2024-03-29T11:53:00.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.001+0000 7f4b44c8b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:00.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.001+0000 7f4b44c8b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:53:00.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.001+0000 7f4b44c8b640 10 monclient: dump: 2024-03-29T11:53:00.010 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:53:00.010 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:00.010 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:00.010 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:00.010 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:00.010 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:00.010 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:53:00.010 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:53:00.010 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:53:00.010 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:00.011 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.002+0000 7f4b47ef1640 10 monclient: get_monmap_and_config success 2024-03-29T11:53:00.011 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.002+0000 7f4b47ef1640 4 set_mon_vals no callback set 2024-03-29T11:53:00.011 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.003+0000 7f4b47ef1640 10 monclient: shutdown 2024-03-29T11:53:00.011 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.003+0000 7f4b47ef1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:53:00.011 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.003+0000 7f4b47ef1640 1 -- 172.21.15.98:0/625840476 >> v1:172.21.15.98:6789/0 conn(0x7f4b4013f650 legacy=0x7f4b400b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:00.011 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.003+0000 7f4b47ef1640 1 -- 172.21.15.98:0/625840476 shutdown_connections 2024-03-29T11:53:00.011 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.003+0000 7f4b47ef1640 1 -- 172.21.15.98:0/625840476 >> 172.21.15.98:0/625840476 conn(0x7f4b40066a00 msgr2=0x7f4b40066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:53:00.011 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.004+0000 7f4b47ef1640 1 -- 172.21.15.98:0/625840476 shutdown_connections 2024-03-29T11:53:00.012 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.004+0000 7f4b47ef1640 1 -- 172.21.15.98:0/625840476 wait complete. 2024-03-29T11:53:00.012 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.005+0000 7f4b47ef1640 10 monclient: build_initial_monmap 2024-03-29T11:53:00.012 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.005+0000 7f4b47ef1640 10 monclient: monmap: 2024-03-29T11:53:00.012 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:53:00.012 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:53:00.012 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:53:00.013 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:53:00.013 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:53:00.013 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:00.013 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:53:00.013 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:53:00.013 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:53:00.013 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:00.014 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.005+0000 7f4b47ef1640 1 librados: starting msgr at 2024-03-29T11:53:00.014 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.005+0000 7f4b47ef1640 1 librados: starting objecter 2024-03-29T11:53:00.014 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.006+0000 7f4b47ef1640 1 Processor -- start 2024-03-29T11:53:00.014 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.006+0000 7f4b47ef1640 1 -- start start 2024-03-29T11:53:00.014 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.006+0000 7f4b47ef1640 1 librados: setting wanted keys 2024-03-29T11:53:00.014 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.006+0000 7f4b47ef1640 1 librados: calling monclient init 2024-03-29T11:53:00.015 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.006+0000 7f4b47ef1640 10 monclient: init 2024-03-29T11:53:00.015 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.006+0000 7f4b47ef1640 10 monclient: _reopen_session rank -1 2024-03-29T11:53:00.015 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.006+0000 7f4b47ef1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:53:00.015 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.006+0000 7f4b47ef1640 10 monclient(hunting): picked mon.noname-b con 0x7f4b400a9d70 addr v1:172.21.15.121:6789/0 2024-03-29T11:53:00.016 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.009+0000 7f4b47ef1640 10 monclient(hunting): picked mon.noname-c con 0x7f4b4013f650 addr v1:172.21.15.98:6790/0 2024-03-29T11:53:00.016 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.009+0000 7f4b47ef1640 10 monclient(hunting): picked mon.noname-a con 0x7f4b401457f0 addr v1:172.21.15.98:6789/0 2024-03-29T11:53:00.016 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.009+0000 7f4b47ef1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b40070250 con 0x7f4b401457f0 2024-03-29T11:53:00.016 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.009+0000 7f4b47ef1640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b40136780 con 0x7f4b400a9d70 2024-03-29T11:53:00.017 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.009+0000 7f4b47ef1640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b40071b90 con 0x7f4b4013f650 2024-03-29T11:53:00.017 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.009+0000 7f4b47ef1640 10 monclient(hunting): _renew_subs 2024-03-29T11:53:00.017 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.009+0000 7f4b47ef1640 10 monclient(hunting): authenticate will time out at 1087.094451s 2024-03-29T11:53:00.017 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.009+0000 7f4b45c8d640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f4b400a9d70 0x7f4b40070700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:55153/0 (socket says 172.21.15.98:55153) 2024-03-29T11:53:00.017 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.009+0000 7f4b45c8d640 1 -- 172.21.15.98:0/2574863853 learned_addr learned my addr 172.21.15.98:0/2574863853 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:53:00.018 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.009+0000 7f4b367fc640 10 client.?.objecter ms_handle_connect 0x7f4b4013f650 2024-03-29T11:53:00.018 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.009+0000 7f4b367fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:53:00.018 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.010+0000 7f4b367fc640 1 -- 172.21.15.98:0/2574863853 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3097468102 0 0) 0x7f4b40071b90 con 0x7f4b4013f650 2024-03-29T11:53:00.018 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.010+0000 7f4b367fc640 10 monclient(hunting): _negotiate 2024-03-29T11:53:00.018 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.010+0000 7f4b367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:00.018 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.010+0000 7f4b367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:00.018 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.010+0000 7f4b367fc640 10 monclient(hunting): my global_id is 4406 2024-03-29T11:53:00.019 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.010+0000 7f4b367fc640 1 -- 172.21.15.98:0/2574863853 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b24004eb0 con 0x7f4b4013f650 2024-03-29T11:53:00.019 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.010+0000 7f4b367fc640 10 client.?.objecter ms_handle_connect 0x7f4b400a9d70 2024-03-29T11:53:00.019 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.010+0000 7f4b367fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:53:00.019 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.010+0000 7f4b367fc640 1 -- 172.21.15.98:0/2574863853 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2004879126 0 0) 0x7f4b24004eb0 con 0x7f4b4013f650 2024-03-29T11:53:00.019 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.010+0000 7f4b367fc640 1 -- 172.21.15.98:0/2574863853 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b40071b90 con 0x7f4b4013f650 2024-03-29T11:53:00.019 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.010+0000 7f4b367fc640 1 -- 172.21.15.98:0/2574863853 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f4b30001a70 con 0x7f4b4013f650 2024-03-29T11:53:00.019 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.010+0000 7f4b367fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:00.019 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.010+0000 7f4b367fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:53:00.019 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.010+0000 7f4b367fc640 10 monclient(hunting): dump: 2024-03-29T11:53:00.019 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:53:00.019 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:00.019 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:00.019 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:00.019 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:00.019 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:00.020 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:53:00.020 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:53:00.020 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:53:00.020 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:00.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.010+0000 7f4b367fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:53:00.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.011+0000 7f4b367fc640 1 -- 172.21.15.98:0/2574863853 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3443451795 0 0) 0x7f4b40136780 con 0x7f4b400a9d70 2024-03-29T11:53:00.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.011+0000 7f4b367fc640 10 monclient(hunting): _negotiate 2024-03-29T11:53:00.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.011+0000 7f4b367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:00.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.011+0000 7f4b367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:00.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.011+0000 7f4b367fc640 10 monclient(hunting): my global_id is 4411 2024-03-29T11:53:00.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.011+0000 7f4b367fc640 1 -- 172.21.15.98:0/2574863853 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b24004eb0 con 0x7f4b400a9d70 2024-03-29T11:53:00.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.011+0000 7f4b367fc640 1 -- 172.21.15.98:0/2574863853 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3848369713 0 0) 0x7f4b40071b90 con 0x7f4b4013f650 2024-03-29T11:53:00.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.011+0000 7f4b367fc640 1 -- 172.21.15.98:0/2574863853 >> v1:172.21.15.121:6789/0 conn(0x7f4b400a9d70 legacy=0x7f4b40070700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:00.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.011+0000 7f4b367fc640 1 -- 172.21.15.98:0/2574863853 >> v1:172.21.15.98:6789/0 conn(0x7f4b401457f0 legacy=0x7f4b40136070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:53:00.022 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.012+0000 7f4b367fc640 10 monclient: _finish_hunting 0 2024-03-29T11:53:00.022 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.012+0000 7f4b367fc640 1 monclient: found mon.c 2024-03-29T11:53:00.022 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.012+0000 7f4b367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:53:00.022 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.012+0000 7f4b367fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:53:00.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.012+0000 7f4b367fc640 1 -- 172.21.15.98:0/2574863853 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b40071e70 con 0x7f4b4013f650 2024-03-29T11:53:00.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.012+0000 7f4b367fc640 10 monclient: _finish_auth 0 2024-03-29T11:53:00.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.012+0000 7f4b367fc640 10 monclient: _check_auth_tickets 2024-03-29T11:53:00.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.012+0000 7f4b367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:53:00.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.012+0000 7f4b47ef1640 5 monclient: authenticate success, global_id 4406 2024-03-29T11:53:00.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.012+0000 7f4b47ef1640 10 monclient: _renew_subs 2024-03-29T11:53:00.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.012+0000 7f4b47ef1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:53:00.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.012+0000 7f4b47ef1640 1 -- 172.21.15.98:0/2574863853 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b401de9d0 con 0x7f4b4013f650 2024-03-29T11:53:00.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.012+0000 7f4b367fc640 1 -- 172.21.15.98:0/2574863853 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b300038a0 con 0x7f4b4013f650 2024-03-29T11:53:00.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.012+0000 7f4b367fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:53:00.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.012+0000 7f4b37fff640 4 set_mon_vals no callback set 2024-03-29T11:53:00.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.013+0000 7f4b47ef1640 10 client.4406.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:53:00.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.013+0000 7f4b367fc640 1 -- 172.21.15.98:0/2574863853 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f4b30003bb0 con 0x7f4b4013f650 2024-03-29T11:53:00.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.013+0000 7f4b367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:00.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.013+0000 7f4b367fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:53:00.024 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.013+0000 7f4b367fc640 10 monclient: dump: 2024-03-29T11:53:00.024 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:53:00.024 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:00.024 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:00.024 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:00.024 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:00.024 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:00.024 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:53:00.024 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:53:00.024 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:53:00.024 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:00.024 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.013+0000 7f4b47ef1640 10 monclient: _renew_subs 2024-03-29T11:53:00.024 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.013+0000 7f4b47ef1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:53:00.024 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.013+0000 7f4b47ef1640 1 -- 172.21.15.98:0/2574863853 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b40070ff0 con 0x7f4b4013f650 2024-03-29T11:53:00.024 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.013+0000 7f4b47ef1640 1 librados: init done 2024-03-29T11:53:00.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.014+0000 7f4b367fc640 1 -- 172.21.15.98:0/2574863853 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f4b30019b80 con 0x7f4b4013f650 2024-03-29T11:53:00.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.015+0000 7f4b45c8d640 10 monclient: get_auth_request con 0x7f4b2406dd30 auth_method 0 2024-03-29T11:53:00.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.016+0000 7f4b367fc640 10 client.4406.objecter ms_handle_connect 0x7f4b2406dd30 2024-03-29T11:53:00.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.016+0000 7f4b367fc640 1 -- 172.21.15.98:0/2574863853 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f4b300537d0 con 0x7f4b4013f650 2024-03-29T11:53:00.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.016+0000 7f4b367fc640 10 client.4406.objecter ms_dispatch 0x7f4b40000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:53:00.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.016+0000 7f4b367fc640 3 client.4406.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:53:00.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.016+0000 7f4b367fc640 3 client.4406.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:53:00.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.016+0000 7f4b367fc640 20 client.4406.objecter dump_active .. 0 homeless 2024-03-29T11:53:00.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.016+0000 7f4b47ef1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:53:00.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.016+0000 7f4b47ef1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:53:00.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.016+0000 7f4b47ef1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:53:00.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.016+0000 7f4b47ef1640 1 -- 172.21.15.98:0/2574863853 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b401deb60 con 0x7f4b4013f650 2024-03-29T11:53:00.033 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.026+0000 7f4b367fc640 1 -- 172.21.15.98:0/2574863853 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4b3004f2e0 con 0x7f4b4013f650 2024-03-29T11:53:00.033 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.026+0000 7f4b367fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:53:00.033 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.026+0000 7f4b367fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:53:00.360 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.353+0000 7f4b47ef1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:53:00.360 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.353+0000 7f4b47ef1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:53:00.360 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.353+0000 7f4b47ef1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:53:00.360 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.353+0000 7f4b47ef1640 1 -- 172.21.15.98:0/2574863853 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b40075f10 con 0x7f4b4013f650 2024-03-29T11:53:00.361 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.354+0000 7f4b367fc640 1 -- 172.21.15.98:0/2574863853 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f4b30051210 con 0x7f4b4013f650 2024-03-29T11:53:00.361 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.354+0000 7f4b367fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:53:00.362 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.354+0000 7f4b367fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:53:00.363 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:53:00.363 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:53:00.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.358+0000 7f4b47ef1640 10 librados: watch_flush enter 2024-03-29T11:53:00.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.358+0000 7f4b47ef1640 10 librados: watch_flush exit 2024-03-29T11:53:00.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.358+0000 7f4b47ef1640 20 client.4406.objecter shutdown clearing up homeless session... 2024-03-29T11:53:00.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.358+0000 7f4b47ef1640 10 client.4406.objecter successfully canceled tick 2024-03-29T11:53:00.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.358+0000 7f4b47ef1640 1 -- 172.21.15.98:0/2574863853 >> v1:172.21.15.98:6816/4065186974 conn(0x7f4b2406dd30 legacy=0x7f4b240701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:00.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.358+0000 7f4b47ef1640 10 monclient: shutdown 2024-03-29T11:53:00.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.358+0000 7f4b47ef1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:53:00.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.358+0000 7f4b47ef1640 1 -- 172.21.15.98:0/2574863853 >> v1:172.21.15.98:6790/0 conn(0x7f4b4013f650 legacy=0x7f4b40135960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:00.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.358+0000 7f4b47ef1640 1 -- 172.21.15.98:0/2574863853 shutdown_connections 2024-03-29T11:53:00.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.358+0000 7f4b47ef1640 1 -- 172.21.15.98:0/2574863853 >> 172.21.15.98:0/2574863853 conn(0x7f4b40066a00 msgr2=0x7f4b400b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:53:00.367 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.359+0000 7f4b47ef1640 1 -- 172.21.15.98:0/2574863853 shutdown_connections 2024-03-29T11:53:00.367 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.359+0000 7f4b47ef1640 1 -- 172.21.15.98:0/2574863853 wait complete. 2024-03-29T11:53:00.367 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:00.359+0000 7f4b47ef1640 1 librados: shutdown 2024-03-29T11:53:00.381 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:53:07.188 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:53:07.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.337+0000 7f1ca359e640 10 monclient: get_monmap_and_config 2024-03-29T11:53:07.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.337+0000 7f1ca359e640 10 monclient: build_initial_monmap 2024-03-29T11:53:07.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.337+0000 7f1ca359e640 10 monclient: monmap: 2024-03-29T11:53:07.344 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:53:07.344 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:07.344 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:53:07.338077+0000 2024-03-29T11:53:07.344 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:53:07.338077+0000 2024-03-29T11:53:07.345 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:53:07.345 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:07.345 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:53:07.345 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:53:07.345 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:53:07.345 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:07.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.339+0000 7f1ca359e640 1 Processor -- start 2024-03-29T11:53:07.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.339+0000 7f1ca359e640 1 -- start start 2024-03-29T11:53:07.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.339+0000 7f1ca359e640 10 monclient: init 2024-03-29T11:53:07.348 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.340+0000 7f1ca359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:53:07.348 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.340+0000 7f1ca359e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:53:07.348 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.340+0000 7f1ca359e640 10 monclient(hunting): picked mon.noname-a con 0x7f1ca40a9750 addr v1:172.21.15.98:6789/0 2024-03-29T11:53:07.348 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.340+0000 7f1ca359e640 10 monclient(hunting): picked mon.noname-c con 0x7f1ca40aad20 addr v1:172.21.15.121:6789/0 2024-03-29T11:53:07.348 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.340+0000 7f1ca359e640 10 monclient(hunting): picked mon.noname-b con 0x7f1ca40a9b30 addr v1:172.21.15.98:6790/0 2024-03-29T11:53:07.348 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.340+0000 7f1ca359e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ca40af4d0 con 0x7f1ca40a9750 2024-03-29T11:53:07.348 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.340+0000 7f1ca359e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ca40af7b0 con 0x7f1ca40aad20 2024-03-29T11:53:07.348 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.340+0000 7f1ca359e640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ca40acab0 con 0x7f1ca40a9b30 2024-03-29T11:53:07.348 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.340+0000 7f1ca359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:53:07.348 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.340+0000 7f1ca359e640 10 monclient(hunting): authenticate will time out at 1094.425936s 2024-03-29T11:53:07.356 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.344+0000 7f1ca2d9d640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f1ca40a9b30 0x7f1ca4149930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:52184/0 (socket says 172.21.15.98:52184) 2024-03-29T11:53:07.356 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.344+0000 7f1ca2d9d640 1 -- 172.21.15.98:0/1875947723 learned_addr learned my addr 172.21.15.98:0/1875947723 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:53:07.356 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.347+0000 7f1ca159a640 1 -- 172.21.15.98:0/1875947723 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 41970218 0 0) 0x7f1ca40acab0 con 0x7f1ca40a9b30 2024-03-29T11:53:07.356 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.347+0000 7f1ca159a640 10 monclient(hunting): _negotiate 2024-03-29T11:53:07.356 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.347+0000 7f1ca159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:07.356 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.347+0000 7f1ca159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:07.356 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.347+0000 7f1ca159a640 10 monclient(hunting): my global_id is 4409 2024-03-29T11:53:07.356 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.347+0000 7f1ca159a640 1 -- 172.21.15.98:0/1875947723 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c8c003520 con 0x7f1ca40a9b30 2024-03-29T11:53:07.356 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.347+0000 7f1ca159a640 1 -- 172.21.15.98:0/1875947723 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3982956637 0 0) 0x7f1ca40af4d0 con 0x7f1ca40a9750 2024-03-29T11:53:07.356 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.347+0000 7f1ca159a640 10 monclient(hunting): _negotiate 2024-03-29T11:53:07.356 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.347+0000 7f1ca159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:07.356 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.347+0000 7f1ca159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:07.356 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.347+0000 7f1ca159a640 10 monclient(hunting): my global_id is 4377 2024-03-29T11:53:07.357 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.347+0000 7f1ca159a640 1 -- 172.21.15.98:0/1875947723 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ca40acab0 con 0x7f1ca40a9750 2024-03-29T11:53:07.357 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.347+0000 7f1ca159a640 1 -- 172.21.15.98:0/1875947723 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3124412750 0 0) 0x7f1ca40af7b0 con 0x7f1ca40aad20 2024-03-29T11:53:07.357 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.347+0000 7f1ca159a640 10 monclient(hunting): _negotiate 2024-03-29T11:53:07.357 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.347+0000 7f1ca159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:07.357 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.347+0000 7f1ca159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:07.357 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.347+0000 7f1ca159a640 10 monclient(hunting): my global_id is 4414 2024-03-29T11:53:07.357 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.348+0000 7f1ca159a640 1 -- 172.21.15.98:0/1875947723 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ca40af4d0 con 0x7f1ca40aad20 2024-03-29T11:53:07.357 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.348+0000 7f1ca159a640 1 -- 172.21.15.98:0/1875947723 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1425762988 0 0) 0x7f1c8c003520 con 0x7f1ca40a9b30 2024-03-29T11:53:07.357 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.348+0000 7f1ca159a640 1 -- 172.21.15.98:0/1875947723 >> v1:172.21.15.121:6789/0 conn(0x7f1ca40aad20 legacy=0x7f1ca4141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:07.357 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.348+0000 7f1ca159a640 1 -- 172.21.15.98:0/1875947723 >> v1:172.21.15.98:6789/0 conn(0x7f1ca40a9750 legacy=0x7f1ca40ac3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:07.357 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.348+0000 7f1ca159a640 10 monclient: _finish_hunting 0 2024-03-29T11:53:07.357 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.348+0000 7f1ca159a640 1 monclient: found mon.noname-b 2024-03-29T11:53:07.357 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.348+0000 7f1ca159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:53:07.357 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.348+0000 7f1ca159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:53:07.357 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.348+0000 7f1ca159a640 1 -- 172.21.15.98:0/1875947723 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca40aba30 con 0x7f1ca40a9b30 2024-03-29T11:53:07.357 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.348+0000 7f1ca159a640 10 monclient: _finish_auth 0 2024-03-29T11:53:07.358 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.348+0000 7f1ca159a640 10 monclient: _check_auth_tickets 2024-03-29T11:53:07.358 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.348+0000 7f1ca159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:53:07.358 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.348+0000 7f1ca159a640 1 -- 172.21.15.98:0/1875947723 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f1c9c001990 con 0x7f1ca40a9b30 2024-03-29T11:53:07.358 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.348+0000 7f1ca159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:07.358 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.348+0000 7f1ca159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:53:07.358 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.348+0000 7f1ca159a640 10 monclient: dump: 2024-03-29T11:53:07.358 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:53:07.358 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:07.358 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:07.358 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:07.358 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:07.358 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:07.358 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:53:07.358 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:53:07.358 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:53:07.358 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:07.359 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.348+0000 7f1ca359e640 5 monclient: authenticate success, global_id 4409 2024-03-29T11:53:07.359 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.348+0000 7f1ca359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:53:07.359 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.348+0000 7f1ca159a640 1 -- 172.21.15.98:0/1875947723 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c9c001af0 con 0x7f1ca40a9b30 2024-03-29T11:53:07.359 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.348+0000 7f1ca159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:53:07.359 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.348+0000 7f1ca359e640 10 monclient: get_monmap_and_config success 2024-03-29T11:53:07.359 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.348+0000 7f1ca359e640 4 set_mon_vals no callback set 2024-03-29T11:53:07.359 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.348+0000 7f1ca159a640 1 -- 172.21.15.98:0/1875947723 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f1c9c001f40 con 0x7f1ca40a9b30 2024-03-29T11:53:07.359 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.348+0000 7f1ca159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:07.359 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.348+0000 7f1ca159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:53:07.359 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.348+0000 7f1ca159a640 10 monclient: dump: 2024-03-29T11:53:07.359 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:53:07.359 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:07.359 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:07.359 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:07.359 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:07.359 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:07.359 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:53:07.360 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:53:07.360 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:53:07.360 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:07.360 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.349+0000 7f1ca359e640 10 monclient: shutdown 2024-03-29T11:53:07.360 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.349+0000 7f1ca359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:53:07.360 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.349+0000 7f1ca359e640 1 -- 172.21.15.98:0/1875947723 >> v1:172.21.15.98:6790/0 conn(0x7f1ca40a9b30 legacy=0x7f1ca4149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:07.361 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.349+0000 7f1ca359e640 1 -- 172.21.15.98:0/1875947723 shutdown_connections 2024-03-29T11:53:07.361 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.349+0000 7f1ca359e640 1 -- 172.21.15.98:0/1875947723 >> 172.21.15.98:0/1875947723 conn(0x7f1ca40673e0 msgr2=0x7f1ca40b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:53:07.361 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.350+0000 7f1ca359e640 1 -- 172.21.15.98:0/1875947723 shutdown_connections 2024-03-29T11:53:07.361 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.352+0000 7f1ca359e640 1 -- 172.21.15.98:0/1875947723 wait complete. 2024-03-29T11:53:07.361 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.353+0000 7f1ca359e640 10 monclient: build_initial_monmap 2024-03-29T11:53:07.362 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.353+0000 7f1ca359e640 10 monclient: monmap: 2024-03-29T11:53:07.362 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:53:07.362 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:53:07.362 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:53:07.362 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:53:07.362 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:53:07.362 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:07.362 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:53:07.362 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:53:07.362 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:53:07.362 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:07.362 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.354+0000 7f1ca359e640 1 librados: starting msgr at 2024-03-29T11:53:07.362 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.354+0000 7f1ca359e640 1 librados: starting objecter 2024-03-29T11:53:07.362 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.354+0000 7f1ca359e640 1 Processor -- start 2024-03-29T11:53:07.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.354+0000 7f1ca359e640 1 -- start start 2024-03-29T11:53:07.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.354+0000 7f1ca359e640 1 librados: setting wanted keys 2024-03-29T11:53:07.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.354+0000 7f1ca359e640 1 librados: calling monclient init 2024-03-29T11:53:07.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.354+0000 7f1ca359e640 10 monclient: init 2024-03-29T11:53:07.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.355+0000 7f1ca359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:53:07.371 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.355+0000 7f1ca359e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:53:07.371 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.355+0000 7f1ca359e640 10 monclient(hunting): picked mon.noname-b con 0x7f1ca40a9750 addr v1:172.21.15.121:6789/0 2024-03-29T11:53:07.371 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.355+0000 7f1ca359e640 10 monclient(hunting): picked mon.noname-c con 0x7f1ca40a9b30 addr v1:172.21.15.98:6790/0 2024-03-29T11:53:07.371 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.355+0000 7f1ca359e640 10 monclient(hunting): picked mon.noname-a con 0x7f1ca40aad20 addr v1:172.21.15.98:6789/0 2024-03-29T11:53:07.371 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.355+0000 7f1ca359e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ca413ee40 con 0x7f1ca40aad20 2024-03-29T11:53:07.371 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.355+0000 7f1ca359e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ca413f120 con 0x7f1ca40a9750 2024-03-29T11:53:07.371 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.355+0000 7f1ca359e640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ca413d760 con 0x7f1ca40a9b30 2024-03-29T11:53:07.371 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.355+0000 7f1ca359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:53:07.371 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.355+0000 7f1ca359e640 10 monclient(hunting): authenticate will time out at 1094.440457s 2024-03-29T11:53:07.371 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.358+0000 7f1ca1d9b640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f1ca40a9b30 0x7f1ca4140960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:54474/0 (socket says 172.21.15.98:54474) 2024-03-29T11:53:07.371 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.358+0000 7f1ca1d9b640 1 -- 172.21.15.98:0/2319238180 learned_addr learned my addr 172.21.15.98:0/2319238180 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:53:07.371 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.359+0000 7f1c92ffd640 10 client.?.objecter ms_handle_connect 0x7f1ca40a9b30 2024-03-29T11:53:07.371 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.359+0000 7f1c92ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:53:07.371 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.359+0000 7f1c92ffd640 10 client.?.objecter ms_handle_connect 0x7f1ca40a9750 2024-03-29T11:53:07.371 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.359+0000 7f1c92ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:53:07.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.359+0000 7f1c92ffd640 1 -- 172.21.15.98:0/2319238180 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3533900776 0 0) 0x7f1ca413d760 con 0x7f1ca40a9b30 2024-03-29T11:53:07.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.359+0000 7f1c92ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:53:07.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.359+0000 7f1c92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:07.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.359+0000 7f1c92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:07.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.359+0000 7f1c92ffd640 10 monclient(hunting): my global_id is 4412 2024-03-29T11:53:07.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.359+0000 7f1c92ffd640 1 -- 172.21.15.98:0/2319238180 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c84004eb0 con 0x7f1ca40a9b30 2024-03-29T11:53:07.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.359+0000 7f1c92ffd640 1 -- 172.21.15.98:0/2319238180 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2793971079 0 0) 0x7f1ca413f120 con 0x7f1ca40a9750 2024-03-29T11:53:07.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.360+0000 7f1c92ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:53:07.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.360+0000 7f1c92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:07.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.360+0000 7f1c92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:07.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.360+0000 7f1c92ffd640 10 monclient(hunting): my global_id is 4417 2024-03-29T11:53:07.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.360+0000 7f1c92ffd640 1 -- 172.21.15.98:0/2319238180 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ca413d760 con 0x7f1ca40a9750 2024-03-29T11:53:07.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.360+0000 7f1c92ffd640 1 -- 172.21.15.98:0/2319238180 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2239739271 0 0) 0x7f1c84004eb0 con 0x7f1ca40a9b30 2024-03-29T11:53:07.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.360+0000 7f1c92ffd640 1 -- 172.21.15.98:0/2319238180 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ca413f120 con 0x7f1ca40a9b30 2024-03-29T11:53:07.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.360+0000 7f1c92ffd640 1 -- 172.21.15.98:0/2319238180 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f1c98001970 con 0x7f1ca40a9b30 2024-03-29T11:53:07.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.360+0000 7f1c92ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:07.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.360+0000 7f1c92ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:53:07.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.360+0000 7f1c92ffd640 10 monclient(hunting): dump: 2024-03-29T11:53:07.373 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:53:07.373 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:07.373 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:07.373 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:07.373 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:07.373 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:07.373 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:53:07.373 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:53:07.373 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:53:07.373 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:07.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.360+0000 7f1c92ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:53:07.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.360+0000 7f1c92ffd640 1 -- 172.21.15.98:0/2319238180 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3454625547 0 0) 0x7f1ca413f120 con 0x7f1ca40a9b30 2024-03-29T11:53:07.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.361+0000 7f1c92ffd640 1 -- 172.21.15.98:0/2319238180 >> v1:172.21.15.121:6789/0 conn(0x7f1ca40a9750 legacy=0x7f1ca41de740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:07.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.361+0000 7f1c92ffd640 1 -- 172.21.15.98:0/2319238180 >> v1:172.21.15.98:6789/0 conn(0x7f1ca40aad20 legacy=0x7f1ca41e6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:07.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.361+0000 7f1c92ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:53:07.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.361+0000 7f1c92ffd640 1 monclient: found mon.c 2024-03-29T11:53:07.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.361+0000 7f1c92ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:53:07.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.361+0000 7f1c92ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:53:07.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.361+0000 7f1c92ffd640 1 -- 172.21.15.98:0/2319238180 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca41dee50 con 0x7f1ca40a9b30 2024-03-29T11:53:07.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.361+0000 7f1c92ffd640 10 monclient: _finish_auth 0 2024-03-29T11:53:07.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.361+0000 7f1c92ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:53:07.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.361+0000 7f1c92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:53:07.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.361+0000 7f1ca359e640 5 monclient: authenticate success, global_id 4412 2024-03-29T11:53:07.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.361+0000 7f1ca359e640 10 monclient: _renew_subs 2024-03-29T11:53:07.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.361+0000 7f1ca359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:53:07.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.361+0000 7f1ca359e640 1 -- 172.21.15.98:0/2319238180 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ca413d3f0 con 0x7f1ca40a9b30 2024-03-29T11:53:07.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.361+0000 7f1ca359e640 10 client.4412.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:53:07.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.361+0000 7f1ca359e640 10 monclient: _renew_subs 2024-03-29T11:53:07.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.361+0000 7f1ca359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:53:07.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.361+0000 7f1ca359e640 1 -- 172.21.15.98:0/2319238180 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ca413d180 con 0x7f1ca40a9b30 2024-03-29T11:53:07.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.361+0000 7f1ca359e640 1 librados: init done 2024-03-29T11:53:07.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.362+0000 7f1c92ffd640 1 -- 172.21.15.98:0/2319238180 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c98001ad0 con 0x7f1ca40a9b30 2024-03-29T11:53:07.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.362+0000 7f1c92ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:53:07.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.362+0000 7f1c92ffd640 1 -- 172.21.15.98:0/2319238180 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f1c980038c0 con 0x7f1ca40a9b30 2024-03-29T11:53:07.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.362+0000 7f1ca0d99640 4 set_mon_vals no callback set 2024-03-29T11:53:07.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.363+0000 7f1ca359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:53:07.376 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.363+0000 7f1ca359e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:53:07.376 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.363+0000 7f1ca359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:53:07.376 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.363+0000 7f1ca359e640 1 -- 172.21.15.98:0/2319238180 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c88005080 con 0x7f1ca40a9b30 2024-03-29T11:53:07.376 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.365+0000 7f1c92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:07.376 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.365+0000 7f1c92ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:53:07.376 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.365+0000 7f1c92ffd640 10 monclient: dump: 2024-03-29T11:53:07.377 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:53:07.377 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:07.377 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:07.377 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:07.377 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:07.377 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:07.377 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:53:07.377 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:53:07.377 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:53:07.377 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:07.377 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.365+0000 7f1c92ffd640 1 -- 172.21.15.98:0/2319238180 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f1c98003b00 con 0x7f1ca40a9b30 2024-03-29T11:53:07.377 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.366+0000 7f1c92ffd640 1 -- 172.21.15.98:0/2319238180 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f1c98084a60 con 0x7f1ca40a9b30 2024-03-29T11:53:07.377 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.366+0000 7f1ca259c640 10 monclient: get_auth_request con 0x7f1c8406dc70 auth_method 0 2024-03-29T11:53:07.377 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.366+0000 7f1c92ffd640 10 client.4412.objecter ms_dispatch 0x7f1ca4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:53:07.377 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.366+0000 7f1c92ffd640 3 client.4412.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:53:07.377 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.366+0000 7f1c92ffd640 3 client.4412.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:53:07.377 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.366+0000 7f1c92ffd640 20 client.4412.objecter dump_active .. 0 homeless 2024-03-29T11:53:07.378 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.366+0000 7f1c92ffd640 10 client.4412.objecter ms_handle_connect 0x7f1c8406dc70 2024-03-29T11:53:07.380 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.373+0000 7f1c92ffd640 1 -- 172.21.15.98:0/2319238180 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1c98050340 con 0x7f1ca40a9b30 2024-03-29T11:53:07.381 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.373+0000 7f1c92ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:53:07.381 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.373+0000 7f1c92ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:53:07.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.690+0000 7f1ca359e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:53:07.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.690+0000 7f1ca359e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:53:07.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.690+0000 7f1ca359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:53:07.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.690+0000 7f1ca359e640 1 -- 172.21.15.98:0/2319238180 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c88005330 con 0x7f1ca40a9b30 2024-03-29T11:53:07.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.692+0000 7f1c92ffd640 1 -- 172.21.15.98:0/2319238180 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f1c98052270 con 0x7f1ca40a9b30 2024-03-29T11:53:07.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.692+0000 7f1c92ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:53:07.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.692+0000 7f1c92ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:53:07.700 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:53:07.700 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:53:07.703 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.696+0000 7f1ca359e640 10 librados: watch_flush enter 2024-03-29T11:53:07.703 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.696+0000 7f1ca359e640 10 librados: watch_flush exit 2024-03-29T11:53:07.704 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.696+0000 7f1ca359e640 20 client.4412.objecter shutdown clearing up homeless session... 2024-03-29T11:53:07.704 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.696+0000 7f1ca359e640 10 client.4412.objecter successfully canceled tick 2024-03-29T11:53:07.704 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.696+0000 7f1ca359e640 1 -- 172.21.15.98:0/2319238180 >> v1:172.21.15.98:6816/4065186974 conn(0x7f1c8406dc70 legacy=0x7f1c84070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:07.704 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.696+0000 7f1ca359e640 10 monclient: shutdown 2024-03-29T11:53:07.704 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.696+0000 7f1ca359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:53:07.704 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.696+0000 7f1ca359e640 1 -- 172.21.15.98:0/2319238180 >> v1:172.21.15.98:6790/0 conn(0x7f1ca40a9b30 legacy=0x7f1ca4140960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:07.705 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.697+0000 7f1ca359e640 1 -- 172.21.15.98:0/2319238180 shutdown_connections 2024-03-29T11:53:07.705 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.697+0000 7f1ca359e640 1 -- 172.21.15.98:0/2319238180 >> 172.21.15.98:0/2319238180 conn(0x7f1ca40673e0 msgr2=0x7f1ca40adad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:53:07.705 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.697+0000 7f1ca359e640 1 -- 172.21.15.98:0/2319238180 shutdown_connections 2024-03-29T11:53:07.705 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.697+0000 7f1ca359e640 1 -- 172.21.15.98:0/2319238180 wait complete. 2024-03-29T11:53:07.705 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:07.697+0000 7f1ca359e640 1 librados: shutdown 2024-03-29T11:53:07.724 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:53:14.531 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:53:14.678 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.670+0000 7f550ef26640 10 monclient: get_monmap_and_config 2024-03-29T11:53:14.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.670+0000 7f550ef26640 10 monclient: build_initial_monmap 2024-03-29T11:53:14.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.670+0000 7f550ef26640 10 monclient: monmap: 2024-03-29T11:53:14.679 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:53:14.679 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:14.679 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:53:14.671295+0000 2024-03-29T11:53:14.679 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:53:14.671295+0000 2024-03-29T11:53:14.679 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:53:14.679 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:14.679 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:53:14.679 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:53:14.679 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:53:14.680 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:14.680 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.671+0000 7f550ef26640 1 Processor -- start 2024-03-29T11:53:14.680 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.671+0000 7f550ef26640 1 -- start start 2024-03-29T11:53:14.680 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.671+0000 7f550ef26640 10 monclient: init 2024-03-29T11:53:14.680 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.672+0000 7f550ef26640 10 monclient: _reopen_session rank -1 2024-03-29T11:53:14.680 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.672+0000 7f550ef26640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:53:14.680 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.672+0000 7f550ef26640 10 monclient(hunting): picked mon.noname-c con 0x7f55080b3f30 addr v1:172.21.15.121:6789/0 2024-03-29T11:53:14.680 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.672+0000 7f550ef26640 10 monclient(hunting): picked mon.noname-b con 0x7f55080ae220 addr v1:172.21.15.98:6790/0 2024-03-29T11:53:14.680 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.672+0000 7f550ef26640 10 monclient(hunting): picked mon.noname-a con 0x7f550813b9d0 addr v1:172.21.15.98:6789/0 2024-03-29T11:53:14.681 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.672+0000 7f550ef26640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5508142490 con 0x7f550813b9d0 2024-03-29T11:53:14.681 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.672+0000 7f550ef26640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55080ac130 con 0x7f55080b3f30 2024-03-29T11:53:14.681 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.672+0000 7f550ef26640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55080ac410 con 0x7f55080ae220 2024-03-29T11:53:14.681 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.672+0000 7f550ef26640 10 monclient(hunting): _renew_subs 2024-03-29T11:53:14.681 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.672+0000 7f550ef26640 10 monclient(hunting): authenticate will time out at 1101.757697s 2024-03-29T11:53:14.681 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.672+0000 7f550d723640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f55080ae220 0x7f550813b2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:56120/0 (socket says 172.21.15.98:56120) 2024-03-29T11:53:14.681 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.672+0000 7f550d723640 1 -- 172.21.15.98:0/2504675692 learned_addr learned my addr 172.21.15.98:0/2504675692 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:53:14.684 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.676+0000 7f550cf22640 1 -- 172.21.15.98:0/2504675692 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2103230507 0 0) 0x7f55080ac410 con 0x7f55080ae220 2024-03-29T11:53:14.684 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.677+0000 7f550cf22640 10 monclient(hunting): _negotiate 2024-03-29T11:53:14.684 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.677+0000 7f550cf22640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:14.684 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.677+0000 7f550cf22640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:14.685 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.677+0000 7f550cf22640 10 monclient(hunting): my global_id is 4415 2024-03-29T11:53:14.685 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.677+0000 7f550cf22640 1 -- 172.21.15.98:0/2504675692 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54f4003520 con 0x7f55080ae220 2024-03-29T11:53:14.686 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.677+0000 7f550cf22640 1 -- 172.21.15.98:0/2504675692 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2497059521 0 0) 0x7f5508142490 con 0x7f550813b9d0 2024-03-29T11:53:14.686 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.677+0000 7f550cf22640 10 monclient(hunting): _negotiate 2024-03-29T11:53:14.686 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.677+0000 7f550cf22640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:14.686 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.677+0000 7f550cf22640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:14.686 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.677+0000 7f550cf22640 10 monclient(hunting): my global_id is 4380 2024-03-29T11:53:14.686 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.677+0000 7f550cf22640 1 -- 172.21.15.98:0/2504675692 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55080ac410 con 0x7f550813b9d0 2024-03-29T11:53:14.686 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.677+0000 7f550cf22640 1 -- 172.21.15.98:0/2504675692 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 497042918 0 0) 0x7f54f4003520 con 0x7f55080ae220 2024-03-29T11:53:14.686 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.677+0000 7f550cf22640 1 -- 172.21.15.98:0/2504675692 >> v1:172.21.15.121:6789/0 conn(0x7f55080b3f30 legacy=0x7f5508141d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:53:14.686 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.677+0000 7f550cf22640 1 -- 172.21.15.98:0/2504675692 >> v1:172.21.15.98:6789/0 conn(0x7f550813b9d0 legacy=0x7f550814bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:14.686 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.677+0000 7f550cf22640 10 monclient: _finish_hunting 0 2024-03-29T11:53:14.686 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.677+0000 7f550cf22640 1 monclient: found mon.noname-b 2024-03-29T11:53:14.687 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.677+0000 7f550cf22640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:53:14.687 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.677+0000 7f550cf22640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:53:14.687 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.677+0000 7f550cf22640 1 -- 172.21.15.98:0/2504675692 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55080afa40 con 0x7f55080ae220 2024-03-29T11:53:14.687 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.677+0000 7f550cf22640 10 monclient: _finish_auth 0 2024-03-29T11:53:14.687 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.677+0000 7f550cf22640 10 monclient: _check_auth_tickets 2024-03-29T11:53:14.687 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.677+0000 7f550cf22640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:53:14.687 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.677+0000 7f550ef26640 5 monclient: authenticate success, global_id 4415 2024-03-29T11:53:14.687 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.677+0000 7f550ef26640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:53:14.687 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.677+0000 7f550cf22640 1 -- 172.21.15.98:0/2504675692 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f54f8001690 con 0x7f55080ae220 2024-03-29T11:53:14.687 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.677+0000 7f550cf22640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:14.687 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.677+0000 7f550cf22640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:53:14.687 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.677+0000 7f550cf22640 10 monclient: dump: 2024-03-29T11:53:14.687 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:53:14.687 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:14.687 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:14.687 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:14.688 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:14.688 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:14.688 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:53:14.688 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:53:14.688 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:53:14.688 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:14.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.677+0000 7f550ef26640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:53:14.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.678+0000 7f550cf22640 1 -- 172.21.15.98:0/2504675692 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54f80018b0 con 0x7f55080ae220 2024-03-29T11:53:14.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.678+0000 7f550cf22640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:53:14.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.678+0000 7f550ef26640 10 monclient: get_monmap_and_config success 2024-03-29T11:53:14.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.678+0000 7f550ef26640 4 set_mon_vals no callback set 2024-03-29T11:53:14.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.678+0000 7f550cf22640 1 -- 172.21.15.98:0/2504675692 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f54f8001bc0 con 0x7f55080ae220 2024-03-29T11:53:14.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.678+0000 7f550cf22640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:14.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.678+0000 7f550cf22640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:53:14.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.678+0000 7f550cf22640 10 monclient: dump: 2024-03-29T11:53:14.689 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:53:14.689 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:14.689 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:14.689 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:14.689 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:14.689 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:14.689 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:53:14.689 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:53:14.689 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:53:14.689 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:14.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.679+0000 7f550ef26640 10 monclient: shutdown 2024-03-29T11:53:14.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.679+0000 7f550ef26640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:53:14.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.679+0000 7f550ef26640 1 -- 172.21.15.98:0/2504675692 >> v1:172.21.15.98:6790/0 conn(0x7f55080ae220 legacy=0x7f550813b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:14.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.681+0000 7f550ef26640 1 -- 172.21.15.98:0/2504675692 shutdown_connections 2024-03-29T11:53:14.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.681+0000 7f550ef26640 1 -- 172.21.15.98:0/2504675692 >> 172.21.15.98:0/2504675692 conn(0x7f55080673e0 msgr2=0x7f55080b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:53:14.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.682+0000 7f550ef26640 1 -- 172.21.15.98:0/2504675692 shutdown_connections 2024-03-29T11:53:14.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.682+0000 7f550ef26640 1 -- 172.21.15.98:0/2504675692 wait complete. 2024-03-29T11:53:14.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.683+0000 7f550ef26640 10 monclient: build_initial_monmap 2024-03-29T11:53:14.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.683+0000 7f550ef26640 10 monclient: monmap: 2024-03-29T11:53:14.692 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:53:14.692 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:53:14.692 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:53:14.692 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:53:14.692 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:53:14.692 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:14.692 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:53:14.692 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:53:14.692 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:53:14.692 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:14.692 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.683+0000 7f550ef26640 1 librados: starting msgr at 2024-03-29T11:53:14.692 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.683+0000 7f550ef26640 1 librados: starting objecter 2024-03-29T11:53:14.692 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.683+0000 7f550ef26640 1 Processor -- start 2024-03-29T11:53:14.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.683+0000 7f550ef26640 1 -- start start 2024-03-29T11:53:14.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.683+0000 7f550ef26640 1 librados: setting wanted keys 2024-03-29T11:53:14.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.683+0000 7f550ef26640 1 librados: calling monclient init 2024-03-29T11:53:14.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.683+0000 7f550ef26640 10 monclient: init 2024-03-29T11:53:14.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.684+0000 7f550ef26640 10 monclient: _reopen_session rank -1 2024-03-29T11:53:14.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.684+0000 7f550ef26640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:53:14.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.684+0000 7f550ef26640 10 monclient(hunting): picked mon.noname-c con 0x7f55080ae220 addr v1:172.21.15.98:6790/0 2024-03-29T11:53:14.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.684+0000 7f550ef26640 10 monclient(hunting): picked mon.noname-a con 0x7f55080b3f30 addr v1:172.21.15.98:6789/0 2024-03-29T11:53:14.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.684+0000 7f550ef26640 10 monclient(hunting): picked mon.noname-b con 0x7f550813b9d0 addr v1:172.21.15.121:6789/0 2024-03-29T11:53:14.695 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.684+0000 7f550ef26640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f550813e110 con 0x7f55080b3f30 2024-03-29T11:53:14.695 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.684+0000 7f550ef26640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5508135390 con 0x7f550813b9d0 2024-03-29T11:53:14.695 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.684+0000 7f550ef26640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f550813ece0 con 0x7f55080ae220 2024-03-29T11:53:14.695 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.684+0000 7f550ef26640 10 monclient(hunting): _renew_subs 2024-03-29T11:53:14.695 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.684+0000 7f550ef26640 10 monclient(hunting): authenticate will time out at 1101.769837s 2024-03-29T11:53:14.695 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.686+0000 7f550df24640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f55080ae220 0x7f550813e5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:57686/0 (socket says 172.21.15.98:57686) 2024-03-29T11:53:14.695 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.686+0000 7f550df24640 1 -- 172.21.15.98:0/3632843511 learned_addr learned my addr 172.21.15.98:0/3632843511 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:53:14.695 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.686+0000 7f54fe7fc640 10 client.?.objecter ms_handle_connect 0x7f55080ae220 2024-03-29T11:53:14.695 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.686+0000 7f54fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:53:14.695 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.688+0000 7f54fe7fc640 1 -- 172.21.15.98:0/3632843511 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 927435509 0 0) 0x7f550813ece0 con 0x7f55080ae220 2024-03-29T11:53:14.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.688+0000 7f54fe7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:53:14.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.688+0000 7f54fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:14.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.688+0000 7f54fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:14.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.688+0000 7f54fe7fc640 10 monclient(hunting): my global_id is 4418 2024-03-29T11:53:14.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.688+0000 7f54fe7fc640 1 -- 172.21.15.98:0/3632843511 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54ec004eb0 con 0x7f55080ae220 2024-03-29T11:53:14.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.688+0000 7f54fe7fc640 1 -- 172.21.15.98:0/3632843511 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1164594466 0 0) 0x7f54ec004eb0 con 0x7f55080ae220 2024-03-29T11:53:14.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.688+0000 7f54fe7fc640 1 -- 172.21.15.98:0/3632843511 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f550813ece0 con 0x7f55080ae220 2024-03-29T11:53:14.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.688+0000 7f54fe7fc640 1 -- 172.21.15.98:0/3632843511 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f5504004400 con 0x7f55080ae220 2024-03-29T11:53:14.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.688+0000 7f54fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:14.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.688+0000 7f54fe7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:53:14.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.688+0000 7f54fe7fc640 10 monclient(hunting): dump: 2024-03-29T11:53:14.696 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:53:14.696 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:14.696 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:14.697 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:14.697 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:14.697 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:14.697 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:53:14.697 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:53:14.697 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:53:14.697 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:14.697 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.688+0000 7f54fe7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:53:14.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.689+0000 7f54fe7fc640 10 client.?.objecter ms_handle_connect 0x7f55080b3f30 2024-03-29T11:53:14.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.689+0000 7f54fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:53:14.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.689+0000 7f54fe7fc640 1 -- 172.21.15.98:0/3632843511 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2164933906 0 0) 0x7f550813e110 con 0x7f55080b3f30 2024-03-29T11:53:14.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.689+0000 7f54fe7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:53:14.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.689+0000 7f54fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:14.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.689+0000 7f54fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:14.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.689+0000 7f54fe7fc640 10 monclient(hunting): my global_id is 4383 2024-03-29T11:53:14.699 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.689+0000 7f54fe7fc640 1 -- 172.21.15.98:0/3632843511 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54ec004eb0 con 0x7f55080b3f30 2024-03-29T11:53:14.699 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.689+0000 7f54fe7fc640 1 -- 172.21.15.98:0/3632843511 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3057634053 0 0) 0x7f54ec004eb0 con 0x7f55080b3f30 2024-03-29T11:53:14.699 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.689+0000 7f54fe7fc640 1 -- 172.21.15.98:0/3632843511 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f550813e110 con 0x7f55080b3f30 2024-03-29T11:53:14.699 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.689+0000 7f54fe7fc640 1 -- 172.21.15.98:0/3632843511 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f54f8001550 con 0x7f55080b3f30 2024-03-29T11:53:14.699 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.689+0000 7f54fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:14.699 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.689+0000 7f54fe7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:53:14.699 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.689+0000 7f54fe7fc640 10 monclient(hunting): dump: 2024-03-29T11:53:14.699 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:53:14.699 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:14.699 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:14.699 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:14.699 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:14.699 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:14.699 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:53:14.700 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:53:14.701 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:53:14.701 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:14.701 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.690+0000 7f54fe7fc640 1 -- 172.21.15.98:0/3632843511 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3902143072 0 0) 0x7f550813ece0 con 0x7f55080ae220 2024-03-29T11:53:14.701 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.690+0000 7f54fe7fc640 1 -- 172.21.15.98:0/3632843511 >> v1:172.21.15.121:6789/0 conn(0x7f550813b9d0 legacy=0x7f5508134c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:14.701 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.690+0000 7f54fe7fc640 1 -- 172.21.15.98:0/3632843511 >> v1:172.21.15.98:6789/0 conn(0x7f55080b3f30 legacy=0x7f5508134570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:14.701 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.690+0000 7f54fe7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:53:14.701 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.690+0000 7f54fe7fc640 1 monclient: found mon.c 2024-03-29T11:53:14.701 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.690+0000 7f54fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:53:14.701 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.690+0000 7f54fe7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:53:14.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.690+0000 7f54fe7fc640 1 -- 172.21.15.98:0/3632843511 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f550806e3b0 con 0x7f55080ae220 2024-03-29T11:53:14.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.690+0000 7f54fe7fc640 10 monclient: _finish_auth 0 2024-03-29T11:53:14.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.690+0000 7f54fe7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:53:14.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.690+0000 7f54fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:53:14.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.690+0000 7f550ef26640 5 monclient: authenticate success, global_id 4418 2024-03-29T11:53:14.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.690+0000 7f550ef26640 10 monclient: _renew_subs 2024-03-29T11:53:14.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.690+0000 7f550ef26640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:53:14.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.690+0000 7f550ef26640 1 -- 172.21.15.98:0/3632843511 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f550806e690 con 0x7f55080ae220 2024-03-29T11:53:14.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.690+0000 7f550ef26640 10 client.4418.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:53:14.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.690+0000 7f550ef26640 10 monclient: _renew_subs 2024-03-29T11:53:14.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.690+0000 7f550ef26640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:53:14.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.690+0000 7f550ef26640 1 -- 172.21.15.98:0/3632843511 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f550813c060 con 0x7f55080ae220 2024-03-29T11:53:14.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.690+0000 7f550ef26640 1 librados: init done 2024-03-29T11:53:14.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.690+0000 7f54fe7fc640 1 -- 172.21.15.98:0/3632843511 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5504004850 con 0x7f55080ae220 2024-03-29T11:53:14.703 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.690+0000 7f54fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:53:14.703 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.690+0000 7f54fe7fc640 1 -- 172.21.15.98:0/3632843511 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f5504004660 con 0x7f55080ae220 2024-03-29T11:53:14.703 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.690+0000 7f54fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:14.703 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.690+0000 7f54fe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:53:14.703 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.690+0000 7f54fe7fc640 10 monclient: dump: 2024-03-29T11:53:14.703 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:53:14.703 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:14.703 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:14.703 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:14.703 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:14.703 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:14.703 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:53:14.703 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:53:14.703 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:53:14.703 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:14.705 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.692+0000 7f54fffff640 4 set_mon_vals no callback set 2024-03-29T11:53:14.705 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.692+0000 7f54fe7fc640 1 -- 172.21.15.98:0/3632843511 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f55040016a0 con 0x7f55080ae220 2024-03-29T11:53:14.705 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.693+0000 7f54fe7fc640 1 -- 172.21.15.98:0/3632843511 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f5504084ec0 con 0x7f55080ae220 2024-03-29T11:53:14.705 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.693+0000 7f54fe7fc640 10 client.4418.objecter ms_dispatch 0x7f5508000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:53:14.705 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.693+0000 7f54fe7fc640 3 client.4418.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:53:14.705 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.693+0000 7f550d723640 10 monclient: get_auth_request con 0x7f54ec06de60 auth_method 0 2024-03-29T11:53:14.705 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.693+0000 7f54fe7fc640 3 client.4418.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:53:14.705 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.693+0000 7f54fe7fc640 20 client.4418.objecter dump_active .. 0 homeless 2024-03-29T11:53:14.705 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.693+0000 7f54fe7fc640 10 client.4418.objecter ms_handle_connect 0x7f54ec06de60 2024-03-29T11:53:14.705 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.693+0000 7f550ef26640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:53:14.705 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.693+0000 7f550ef26640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:53:14.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.693+0000 7f550ef26640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:53:14.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.693+0000 7f550ef26640 1 -- 172.21.15.98:0/3632843511 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54f0005080 con 0x7f55080ae220 2024-03-29T11:53:14.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.703+0000 7f54fe7fc640 1 -- 172.21.15.98:0/3632843511 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f55040505d0 con 0x7f55080ae220 2024-03-29T11:53:14.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.703+0000 7f54fe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:53:14.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:14.703+0000 7f54fe7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:53:15.029 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:15.021+0000 7f550ef26640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:53:15.029 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:15.021+0000 7f550ef26640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:53:15.029 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:15.021+0000 7f550ef26640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:53:15.029 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:15.021+0000 7f550ef26640 1 -- 172.21.15.98:0/3632843511 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54f0005330 con 0x7f55080ae220 2024-03-29T11:53:15.030 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:15.023+0000 7f54fe7fc640 1 -- 172.21.15.98:0/3632843511 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f55040526d0 con 0x7f55080ae220 2024-03-29T11:53:15.030 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:15.023+0000 7f54fe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:53:15.030 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:15.023+0000 7f54fe7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:53:15.031 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:53:15.031 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:53:15.035 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:15.026+0000 7f550ef26640 10 librados: watch_flush enter 2024-03-29T11:53:15.035 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:15.027+0000 7f550ef26640 10 librados: watch_flush exit 2024-03-29T11:53:15.035 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:15.027+0000 7f550ef26640 20 client.4418.objecter shutdown clearing up homeless session... 2024-03-29T11:53:15.035 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:15.027+0000 7f550ef26640 10 client.4418.objecter successfully canceled tick 2024-03-29T11:53:15.035 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:15.027+0000 7f550ef26640 1 -- 172.21.15.98:0/3632843511 >> v1:172.21.15.98:6816/4065186974 conn(0x7f54ec06de60 legacy=0x7f54ec070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:15.035 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:15.027+0000 7f550ef26640 10 monclient: shutdown 2024-03-29T11:53:15.035 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:15.027+0000 7f550ef26640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:53:15.035 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:15.027+0000 7f550ef26640 1 -- 172.21.15.98:0/3632843511 >> v1:172.21.15.98:6790/0 conn(0x7f55080ae220 legacy=0x7f550813e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:15.035 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:15.027+0000 7f550ef26640 1 -- 172.21.15.98:0/3632843511 shutdown_connections 2024-03-29T11:53:15.035 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:15.027+0000 7f550ef26640 1 -- 172.21.15.98:0/3632843511 >> 172.21.15.98:0/3632843511 conn(0x7f55080673e0 msgr2=0x7f550813b090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:53:15.035 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:15.027+0000 7f550ef26640 1 -- 172.21.15.98:0/3632843511 shutdown_connections 2024-03-29T11:53:15.035 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:15.027+0000 7f550ef26640 1 -- 172.21.15.98:0/3632843511 wait complete. 2024-03-29T11:53:15.035 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:15.027+0000 7f550ef26640 1 librados: shutdown 2024-03-29T11:53:15.053 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:53:21.861 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:53:22.008 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.000+0000 7efe51927640 10 monclient: get_monmap_and_config 2024-03-29T11:53:22.008 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.000+0000 7efe51927640 10 monclient: build_initial_monmap 2024-03-29T11:53:22.008 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.000+0000 7efe51927640 10 monclient: monmap: 2024-03-29T11:53:22.008 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:53:22.008 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:22.008 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:53:22.001350+0000 2024-03-29T11:53:22.008 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:53:22.001350+0000 2024-03-29T11:53:22.008 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:53:22.008 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:22.009 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:53:22.009 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:53:22.009 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:53:22.009 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:22.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.002+0000 7efe51927640 1 Processor -- start 2024-03-29T11:53:22.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.002+0000 7efe51927640 1 -- start start 2024-03-29T11:53:22.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.002+0000 7efe51927640 10 monclient: init 2024-03-29T11:53:22.013 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.003+0000 7efe51927640 10 monclient: _reopen_session rank -1 2024-03-29T11:53:22.013 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.003+0000 7efe51927640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:53:22.013 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.004+0000 7efe51927640 10 monclient(hunting): picked mon.noname-a con 0x7efe44006a70 addr v1:172.21.15.98:6789/0 2024-03-29T11:53:22.013 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.004+0000 7efe51927640 10 monclient(hunting): picked mon.noname-b con 0x7efe44033060 addr v1:172.21.15.98:6790/0 2024-03-29T11:53:22.013 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.004+0000 7efe4bfff640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7efe44033060 0x7efe440c1760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:41902/0 (socket says 172.21.15.98:41902) 2024-03-29T11:53:22.013 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.004+0000 7efe4bfff640 1 -- 172.21.15.98:0/225413781 learned_addr learned my addr 172.21.15.98:0/225413781 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:53:22.013 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.004+0000 7efe51927640 10 monclient(hunting): picked mon.noname-c con 0x7efe440c1e70 addr v1:172.21.15.121:6789/0 2024-03-29T11:53:22.013 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.004+0000 7efe51927640 1 -- 172.21.15.98:0/225413781 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe4402cfb0 con 0x7efe44006a70 2024-03-29T11:53:22.013 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.004+0000 7efe51927640 1 -- 172.21.15.98:0/225413781 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe4402ae70 con 0x7efe440c1e70 2024-03-29T11:53:22.014 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.004+0000 7efe51927640 1 -- 172.21.15.98:0/225413781 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe440c8110 con 0x7efe44033060 2024-03-29T11:53:22.014 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.004+0000 7efe51927640 10 monclient(hunting): _renew_subs 2024-03-29T11:53:22.014 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.004+0000 7efe51927640 10 monclient(hunting): authenticate will time out at 1109.089907s 2024-03-29T11:53:22.014 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.005+0000 7efe4b7fe640 1 -- 172.21.15.98:0/225413781 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 528198978 0 0) 0x7efe440c8110 con 0x7efe44033060 2024-03-29T11:53:22.014 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.005+0000 7efe4b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:53:22.014 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.005+0000 7efe4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:22.014 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.005+0000 7efe4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:22.014 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.005+0000 7efe4b7fe640 10 monclient(hunting): my global_id is 4421 2024-03-29T11:53:22.014 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.005+0000 7efe4b7fe640 1 -- 172.21.15.98:0/225413781 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe3c003520 con 0x7efe44033060 2024-03-29T11:53:22.015 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.007+0000 7efe4b7fe640 1 -- 172.21.15.98:0/225413781 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2775456996 0 0) 0x7efe3c003520 con 0x7efe44033060 2024-03-29T11:53:22.015 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.007+0000 7efe4b7fe640 1 -- 172.21.15.98:0/225413781 >> v1:172.21.15.121:6789/0 conn(0x7efe440c1e70 legacy=0x7efe440cfda0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:53:22.015 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.008+0000 7efe4b7fe640 1 -- 172.21.15.98:0/225413781 >> v1:172.21.15.98:6789/0 conn(0x7efe44006a70 legacy=0x7efe440c7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:22.019 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.009+0000 7efe4b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:53:22.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.009+0000 7efe4b7fe640 1 monclient: found mon.noname-b 2024-03-29T11:53:22.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.009+0000 7efe4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:53:22.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.009+0000 7efe4b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:53:22.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.009+0000 7efe4b7fe640 1 -- 172.21.15.98:0/225413781 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe4402d200 con 0x7efe44033060 2024-03-29T11:53:22.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.009+0000 7efe4b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:53:22.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.009+0000 7efe4b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:53:22.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.009+0000 7efe4b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:53:22.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.009+0000 7efe4b7fe640 1 -- 172.21.15.98:0/225413781 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7efe40001690 con 0x7efe44033060 2024-03-29T11:53:22.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.010+0000 7efe4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:22.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.010+0000 7efe4b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:53:22.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.010+0000 7efe4b7fe640 10 monclient: dump: 2024-03-29T11:53:22.020 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:53:22.020 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:22.020 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:22.021 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:22.021 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:22.021 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:22.021 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:53:22.021 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:53:22.021 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:53:22.021 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:22.021 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.010+0000 7efe51927640 5 monclient: authenticate success, global_id 4421 2024-03-29T11:53:22.021 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.010+0000 7efe51927640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:53:22.021 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.011+0000 7efe4b7fe640 1 -- 172.21.15.98:0/225413781 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe40001860 con 0x7efe44033060 2024-03-29T11:53:22.021 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.012+0000 7efe4b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:53:22.021 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.012+0000 7efe4b7fe640 1 -- 172.21.15.98:0/225413781 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7efe40001c70 con 0x7efe44033060 2024-03-29T11:53:22.021 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.012+0000 7efe51927640 10 monclient: get_monmap_and_config success 2024-03-29T11:53:22.021 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.012+0000 7efe51927640 4 set_mon_vals no callback set 2024-03-29T11:53:22.022 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.012+0000 7efe4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:22.022 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.012+0000 7efe4b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:53:22.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.012+0000 7efe4b7fe640 10 monclient: dump: 2024-03-29T11:53:22.023 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:53:22.023 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:22.023 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:22.023 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:22.023 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:22.023 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:22.023 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:53:22.023 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:53:22.023 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:53:22.023 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:22.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.012+0000 7efe51927640 10 monclient: shutdown 2024-03-29T11:53:22.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.013+0000 7efe51927640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:53:22.024 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.013+0000 7efe51927640 1 -- 172.21.15.98:0/225413781 >> v1:172.21.15.98:6790/0 conn(0x7efe44033060 legacy=0x7efe440c1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:22.024 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.013+0000 7efe51927640 1 -- 172.21.15.98:0/225413781 shutdown_connections 2024-03-29T11:53:22.024 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.013+0000 7efe51927640 1 -- 172.21.15.98:0/225413781 >> 172.21.15.98:0/225413781 conn(0x7efe440058f0 msgr2=0x7efe44005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:53:22.024 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.015+0000 7efe51927640 1 -- 172.21.15.98:0/225413781 shutdown_connections 2024-03-29T11:53:22.024 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.015+0000 7efe51927640 1 -- 172.21.15.98:0/225413781 wait complete. 2024-03-29T11:53:22.029 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.016+0000 7efe51927640 10 monclient: build_initial_monmap 2024-03-29T11:53:22.029 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.016+0000 7efe51927640 10 monclient: monmap: 2024-03-29T11:53:22.029 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:53:22.029 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:53:22.029 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:53:22.030 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:53:22.030 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:53:22.030 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:22.030 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:53:22.030 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:53:22.030 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:53:22.030 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:22.031 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.016+0000 7efe51927640 1 librados: starting msgr at 2024-03-29T11:53:22.031 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.016+0000 7efe51927640 1 librados: starting objecter 2024-03-29T11:53:22.031 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.017+0000 7efe51927640 1 Processor -- start 2024-03-29T11:53:22.031 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.017+0000 7efe51927640 1 -- start start 2024-03-29T11:53:22.031 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.017+0000 7efe51927640 1 librados: setting wanted keys 2024-03-29T11:53:22.031 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.017+0000 7efe51927640 1 librados: calling monclient init 2024-03-29T11:53:22.031 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.017+0000 7efe51927640 10 monclient: init 2024-03-29T11:53:22.031 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.018+0000 7efe51927640 10 monclient: _reopen_session rank -1 2024-03-29T11:53:22.031 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.018+0000 7efe51927640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:53:22.031 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.018+0000 7efe51927640 10 monclient(hunting): picked mon.noname-b con 0x7efe44006a70 addr v1:172.21.15.121:6789/0 2024-03-29T11:53:22.031 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.018+0000 7efe51927640 10 monclient(hunting): picked mon.noname-a con 0x7efe440c1e70 addr v1:172.21.15.98:6789/0 2024-03-29T11:53:22.031 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.018+0000 7efe51927640 10 monclient(hunting): picked mon.noname-c con 0x7efe440c65c0 addr v1:172.21.15.98:6790/0 2024-03-29T11:53:22.031 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.018+0000 7efe51927640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe440c38f0 con 0x7efe440c1e70 2024-03-29T11:53:22.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.018+0000 7efe51927640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe440c5360 con 0x7efe44006a70 2024-03-29T11:53:22.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.018+0000 7efe51927640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe4414d330 con 0x7efe440c65c0 2024-03-29T11:53:22.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.018+0000 7efe51927640 10 monclient(hunting): _renew_subs 2024-03-29T11:53:22.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.018+0000 7efe51927640 10 monclient(hunting): authenticate will time out at 1109.103755s 2024-03-29T11:53:22.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.019+0000 7efe51126640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7efe440c65c0 0x7efe440b8180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:43904/0 (socket says 172.21.15.98:43904) 2024-03-29T11:53:22.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.019+0000 7efe51126640 1 -- 172.21.15.98:0/1807491995 learned_addr learned my addr 172.21.15.98:0/1807491995 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:53:22.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.019+0000 7efe497fa640 10 client.?.objecter ms_handle_connect 0x7efe440c65c0 2024-03-29T11:53:22.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.020+0000 7efe497fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:53:22.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.020+0000 7efe497fa640 1 -- 172.21.15.98:0/1807491995 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2557473982 0 0) 0x7efe4414d330 con 0x7efe440c65c0 2024-03-29T11:53:22.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.020+0000 7efe497fa640 10 monclient(hunting): _negotiate 2024-03-29T11:53:22.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.020+0000 7efe497fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:22.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.020+0000 7efe497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:22.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.020+0000 7efe497fa640 10 monclient(hunting): my global_id is 4424 2024-03-29T11:53:22.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.020+0000 7efe497fa640 1 -- 172.21.15.98:0/1807491995 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe30004eb0 con 0x7efe440c65c0 2024-03-29T11:53:22.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.022+0000 7efe497fa640 10 client.?.objecter ms_handle_connect 0x7efe44006a70 2024-03-29T11:53:22.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.022+0000 7efe497fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:53:22.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.022+0000 7efe497fa640 1 -- 172.21.15.98:0/1807491995 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 724284274 0 0) 0x7efe30004eb0 con 0x7efe440c65c0 2024-03-29T11:53:22.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.022+0000 7efe497fa640 1 -- 172.21.15.98:0/1807491995 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe4414d330 con 0x7efe440c65c0 2024-03-29T11:53:22.038 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.022+0000 7efe497fa640 1 -- 172.21.15.98:0/1807491995 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7efe4c04fdd0 con 0x7efe440c65c0 2024-03-29T11:53:22.038 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.022+0000 7efe497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:22.038 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.022+0000 7efe497fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:53:22.038 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.022+0000 7efe497fa640 10 monclient(hunting): dump: 2024-03-29T11:53:22.038 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:53:22.038 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:22.039 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:22.039 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:22.039 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:22.039 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:22.039 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:53:22.039 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:53:22.039 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:53:22.039 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:22.039 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.022+0000 7efe497fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:53:22.039 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.022+0000 7efe497fa640 1 -- 172.21.15.98:0/1807491995 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 507429558 0 0) 0x7efe440c5360 con 0x7efe44006a70 2024-03-29T11:53:22.039 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.022+0000 7efe497fa640 10 monclient(hunting): _negotiate 2024-03-29T11:53:22.039 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.022+0000 7efe497fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:22.039 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.022+0000 7efe497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:22.040 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.022+0000 7efe497fa640 10 monclient(hunting): my global_id is 4420 2024-03-29T11:53:22.040 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.023+0000 7efe497fa640 1 -- 172.21.15.98:0/1807491995 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe30004eb0 con 0x7efe44006a70 2024-03-29T11:53:22.040 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.023+0000 7efe497fa640 10 client.?.objecter ms_handle_connect 0x7efe440c1e70 2024-03-29T11:53:22.040 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.023+0000 7efe497fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:53:22.040 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.023+0000 7efe497fa640 1 -- 172.21.15.98:0/1807491995 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3319422944 0 0) 0x7efe440c38f0 con 0x7efe440c1e70 2024-03-29T11:53:22.040 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.023+0000 7efe497fa640 10 monclient(hunting): _negotiate 2024-03-29T11:53:22.040 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.023+0000 7efe497fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:22.040 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.023+0000 7efe497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:22.040 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.023+0000 7efe497fa640 10 monclient(hunting): my global_id is 4389 2024-03-29T11:53:22.040 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.023+0000 7efe497fa640 1 -- 172.21.15.98:0/1807491995 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe440c5360 con 0x7efe440c1e70 2024-03-29T11:53:22.040 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.023+0000 7efe497fa640 1 -- 172.21.15.98:0/1807491995 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3267581454 0 0) 0x7efe4414d330 con 0x7efe440c65c0 2024-03-29T11:53:22.040 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.023+0000 7efe497fa640 1 -- 172.21.15.98:0/1807491995 >> v1:172.21.15.121:6789/0 conn(0x7efe44006a70 legacy=0x7efe440c31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:22.040 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.023+0000 7efe497fa640 1 -- 172.21.15.98:0/1807491995 >> v1:172.21.15.98:6789/0 conn(0x7efe440c1e70 legacy=0x7efe440c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:22.040 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.023+0000 7efe497fa640 10 monclient: _finish_hunting 0 2024-03-29T11:53:22.040 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.024+0000 7efe497fa640 1 monclient: found mon.c 2024-03-29T11:53:22.041 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.024+0000 7efe497fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:53:22.041 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.024+0000 7efe497fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:53:22.041 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.024+0000 7efe497fa640 1 -- 172.21.15.98:0/1807491995 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe440c3b40 con 0x7efe440c65c0 2024-03-29T11:53:22.041 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.024+0000 7efe497fa640 10 monclient: _finish_auth 0 2024-03-29T11:53:22.041 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.024+0000 7efe497fa640 10 monclient: _check_auth_tickets 2024-03-29T11:53:22.041 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.024+0000 7efe497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:53:22.041 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.024+0000 7efe497fa640 1 -- 172.21.15.98:0/1807491995 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe4c04f400 con 0x7efe440c65c0 2024-03-29T11:53:22.041 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.024+0000 7efe497fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:53:22.041 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.024+0000 7efe51927640 5 monclient: authenticate success, global_id 4424 2024-03-29T11:53:22.041 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.024+0000 7efe51927640 10 monclient: _renew_subs 2024-03-29T11:53:22.041 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.024+0000 7efe51927640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:53:22.041 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.024+0000 7efe51927640 1 -- 172.21.15.98:0/1807491995 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe440c3e20 con 0x7efe440c65c0 2024-03-29T11:53:22.041 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.024+0000 7efe51927640 10 client.4424.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:53:22.041 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.024+0000 7efe51927640 10 monclient: _renew_subs 2024-03-29T11:53:22.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.024+0000 7efe51927640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:53:22.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.024+0000 7efe51927640 1 -- 172.21.15.98:0/1807491995 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efe440c4520 con 0x7efe440c65c0 2024-03-29T11:53:22.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.024+0000 7efe51927640 1 librados: init done 2024-03-29T11:53:22.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.024+0000 7efe4affd640 4 set_mon_vals no callback set 2024-03-29T11:53:22.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.024+0000 7efe497fa640 1 -- 172.21.15.98:0/1807491995 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7efe4c075f10 con 0x7efe440c65c0 2024-03-29T11:53:22.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.024+0000 7efe497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:22.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.025+0000 7efe497fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:53:22.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.025+0000 7efe497fa640 10 monclient: dump: 2024-03-29T11:53:22.042 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:53:22.042 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:22.042 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:22.042 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:22.042 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:22.042 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:22.043 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:53:22.043 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:53:22.043 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:53:22.043 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:22.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.026+0000 7efe51927640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:53:22.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.026+0000 7efe51927640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:53:22.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.026+0000 7efe51927640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:53:22.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.026+0000 7efe51927640 1 -- 172.21.15.98:0/1807491995 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe2c005080 con 0x7efe440c65c0 2024-03-29T11:53:22.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.027+0000 7efe497fa640 1 -- 172.21.15.98:0/1807491995 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7efe4c067bf0 con 0x7efe440c65c0 2024-03-29T11:53:22.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.028+0000 7efe497fa640 1 -- 172.21.15.98:0/1807491995 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7efe4c0ff530 con 0x7efe440c65c0 2024-03-29T11:53:22.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.028+0000 7efe497fa640 10 client.4424.objecter ms_dispatch 0x7efe4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:53:22.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.028+0000 7efe497fa640 3 client.4424.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:53:22.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.028+0000 7efe497fa640 3 client.4424.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:53:22.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.028+0000 7efe497fa640 20 client.4424.objecter dump_active .. 0 homeless 2024-03-29T11:53:22.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.032+0000 7efe50925640 10 monclient: get_auth_request con 0x7efe3006d980 auth_method 0 2024-03-29T11:53:22.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.032+0000 7efe497fa640 10 client.4424.objecter ms_handle_connect 0x7efe3006d980 2024-03-29T11:53:22.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.036+0000 7efe497fa640 1 -- 172.21.15.98:0/1807491995 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efe4c0cae10 con 0x7efe440c65c0 2024-03-29T11:53:22.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.036+0000 7efe497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:53:22.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.036+0000 7efe497fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:53:22.358 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.351+0000 7efe51927640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:53:22.359 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.351+0000 7efe51927640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:53:22.359 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.351+0000 7efe51927640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:53:22.359 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.351+0000 7efe51927640 1 -- 172.21.15.98:0/1807491995 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe2c005330 con 0x7efe440c65c0 2024-03-29T11:53:22.359 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.351+0000 7efe497fa640 1 -- 172.21.15.98:0/1807491995 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7efe4c0ccd40 con 0x7efe440c65c0 2024-03-29T11:53:22.359 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.351+0000 7efe497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:53:22.359 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.351+0000 7efe497fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:53:22.360 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:53:22.360 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:53:22.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.355+0000 7efe51927640 10 librados: watch_flush enter 2024-03-29T11:53:22.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.355+0000 7efe51927640 10 librados: watch_flush exit 2024-03-29T11:53:22.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.355+0000 7efe51927640 20 client.4424.objecter shutdown clearing up homeless session... 2024-03-29T11:53:22.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.355+0000 7efe51927640 10 client.4424.objecter successfully canceled tick 2024-03-29T11:53:22.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.355+0000 7efe51927640 1 -- 172.21.15.98:0/1807491995 >> v1:172.21.15.98:6816/4065186974 conn(0x7efe3006d980 legacy=0x7efe3006fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:22.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.355+0000 7efe51927640 10 monclient: shutdown 2024-03-29T11:53:22.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.355+0000 7efe51927640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:53:22.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.355+0000 7efe51927640 1 -- 172.21.15.98:0/1807491995 >> v1:172.21.15.98:6790/0 conn(0x7efe440c65c0 legacy=0x7efe440b8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:22.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.355+0000 7efe51927640 1 -- 172.21.15.98:0/1807491995 shutdown_connections 2024-03-29T11:53:22.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.355+0000 7efe51927640 1 -- 172.21.15.98:0/1807491995 >> 172.21.15.98:0/1807491995 conn(0x7efe440058f0 msgr2=0x7efe44033f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:53:22.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.355+0000 7efe51927640 1 -- 172.21.15.98:0/1807491995 shutdown_connections 2024-03-29T11:53:22.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.356+0000 7efe51927640 1 -- 172.21.15.98:0/1807491995 wait complete. 2024-03-29T11:53:22.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:22.356+0000 7efe51927640 1 librados: shutdown 2024-03-29T11:53:22.381 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:53:29.189 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:53:29.339 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.330+0000 7f23b2fa5640 10 monclient: get_monmap_and_config 2024-03-29T11:53:29.339 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.330+0000 7f23b2fa5640 10 monclient: build_initial_monmap 2024-03-29T11:53:29.341 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.330+0000 7f23b2fa5640 10 monclient: monmap: 2024-03-29T11:53:29.341 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:53:29.341 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:29.341 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:53:29.331064+0000 2024-03-29T11:53:29.341 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:53:29.331064+0000 2024-03-29T11:53:29.341 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:53:29.341 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:29.341 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:53:29.341 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:53:29.341 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:53:29.342 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:29.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.331+0000 7f23b2fa5640 1 Processor -- start 2024-03-29T11:53:29.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.331+0000 7f23b2fa5640 1 -- start start 2024-03-29T11:53:29.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.331+0000 7f23b2fa5640 10 monclient: init 2024-03-29T11:53:29.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.332+0000 7f23b2fa5640 10 monclient: _reopen_session rank -1 2024-03-29T11:53:29.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.332+0000 7f23b2fa5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:53:29.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.332+0000 7f23b2fa5640 10 monclient(hunting): picked mon.noname-c con 0x7f23ac0a9930 addr v1:172.21.15.121:6789/0 2024-03-29T11:53:29.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.332+0000 7f23b2fa5640 10 monclient(hunting): picked mon.noname-b con 0x7f23ac149dc0 addr v1:172.21.15.98:6790/0 2024-03-29T11:53:29.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.332+0000 7f23b2fa5640 10 monclient(hunting): picked mon.noname-a con 0x7f23ac14ff40 addr v1:172.21.15.98:6789/0 2024-03-29T11:53:29.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.332+0000 7f23b2fa5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23ac0aae60 con 0x7f23ac14ff40 2024-03-29T11:53:29.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.332+0000 7f23b2fa5640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23ac0b4310 con 0x7f23ac0a9930 2024-03-29T11:53:29.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.332+0000 7f23b2fa5640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23ac0afdd0 con 0x7f23ac149dc0 2024-03-29T11:53:29.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.332+0000 7f23b2fa5640 10 monclient(hunting): _renew_subs 2024-03-29T11:53:29.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.332+0000 7f23b2fa5640 10 monclient(hunting): authenticate will time out at 1116.417372s 2024-03-29T11:53:29.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.334+0000 7f23abfff640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f23ac149dc0 0x7f23ac14a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:48028/0 (socket says 172.21.15.98:48028) 2024-03-29T11:53:29.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.334+0000 7f23abfff640 1 -- 172.21.15.98:0/998130384 learned_addr learned my addr 172.21.15.98:0/998130384 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:53:29.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.336+0000 7f23ab7fe640 1 -- 172.21.15.98:0/998130384 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1401446539 0 0) 0x7f23ac0afdd0 con 0x7f23ac149dc0 2024-03-29T11:53:29.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.336+0000 7f23ab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:53:29.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.336+0000 7f23ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:29.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.336+0000 7f23ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:29.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.336+0000 7f23ab7fe640 10 monclient(hunting): my global_id is 4427 2024-03-29T11:53:29.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.336+0000 7f23ab7fe640 1 -- 172.21.15.98:0/998130384 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2398003520 con 0x7f23ac149dc0 2024-03-29T11:53:29.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.336+0000 7f23ab7fe640 1 -- 172.21.15.98:0/998130384 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3224459762 0 0) 0x7f2398003520 con 0x7f23ac149dc0 2024-03-29T11:53:29.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.336+0000 7f23ab7fe640 1 -- 172.21.15.98:0/998130384 >> v1:172.21.15.121:6789/0 conn(0x7f23ac0a9930 legacy=0x7f23ac143a30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:53:29.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.337+0000 7f23ab7fe640 1 -- 172.21.15.98:0/998130384 >> v1:172.21.15.98:6789/0 conn(0x7f23ac14ff40 legacy=0x7f23ac152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:29.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.337+0000 7f23ab7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:53:29.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.337+0000 7f23ab7fe640 1 monclient: found mon.noname-b 2024-03-29T11:53:29.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.337+0000 7f23ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:53:29.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.337+0000 7f23ab7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:53:29.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.337+0000 7f23ab7fe640 1 -- 172.21.15.98:0/998130384 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23ac14a950 con 0x7f23ac149dc0 2024-03-29T11:53:29.346 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.337+0000 7f23ab7fe640 10 monclient: _finish_auth 0 2024-03-29T11:53:29.346 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.337+0000 7f23ab7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:53:29.346 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.337+0000 7f23ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:53:29.346 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.337+0000 7f23ab7fe640 1 -- 172.21.15.98:0/998130384 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f239c001690 con 0x7f23ac149dc0 2024-03-29T11:53:29.346 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.337+0000 7f23ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:29.346 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.337+0000 7f23ab7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:53:29.346 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.337+0000 7f23ab7fe640 10 monclient: dump: 2024-03-29T11:53:29.346 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:53:29.346 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:29.346 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:29.346 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:29.346 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:29.347 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:29.347 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:53:29.347 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:53:29.347 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:53:29.347 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:29.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.337+0000 7f23b2fa5640 5 monclient: authenticate success, global_id 4427 2024-03-29T11:53:29.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.337+0000 7f23b2fa5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:53:29.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.338+0000 7f23ab7fe640 1 -- 172.21.15.98:0/998130384 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f239c001860 con 0x7f23ac149dc0 2024-03-29T11:53:29.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.338+0000 7f23ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:53:29.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.338+0000 7f23ab7fe640 1 -- 172.21.15.98:0/998130384 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f239c001cb0 con 0x7f23ac149dc0 2024-03-29T11:53:29.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.338+0000 7f23ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:29.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.338+0000 7f23ab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:53:29.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.338+0000 7f23ab7fe640 10 monclient: dump: 2024-03-29T11:53:29.347 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:53:29.347 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:29.347 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:29.348 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:29.348 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:29.348 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:29.348 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:53:29.348 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:53:29.348 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:53:29.348 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:29.348 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.338+0000 7f23b2fa5640 10 monclient: get_monmap_and_config success 2024-03-29T11:53:29.348 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.338+0000 7f23b2fa5640 4 set_mon_vals no callback set 2024-03-29T11:53:29.349 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.338+0000 7f23b2fa5640 10 monclient: shutdown 2024-03-29T11:53:29.349 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.338+0000 7f23b2fa5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:53:29.349 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.338+0000 7f23b2fa5640 1 -- 172.21.15.98:0/998130384 >> v1:172.21.15.98:6790/0 conn(0x7f23ac149dc0 legacy=0x7f23ac14a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:29.349 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.339+0000 7f23b2fa5640 1 -- 172.21.15.98:0/998130384 shutdown_connections 2024-03-29T11:53:29.349 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.339+0000 7f23b2fa5640 1 -- 172.21.15.98:0/998130384 >> 172.21.15.98:0/998130384 conn(0x7f23ac0b35e0 msgr2=0x7f23ac0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:53:29.349 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.339+0000 7f23b2fa5640 1 -- 172.21.15.98:0/998130384 shutdown_connections 2024-03-29T11:53:29.349 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.341+0000 7f23b2fa5640 1 -- 172.21.15.98:0/998130384 wait complete. 2024-03-29T11:53:29.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.342+0000 7f23b2fa5640 10 monclient: build_initial_monmap 2024-03-29T11:53:29.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.342+0000 7f23b2fa5640 10 monclient: monmap: 2024-03-29T11:53:29.350 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:53:29.351 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:53:29.351 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:53:29.351 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:53:29.351 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:53:29.351 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:29.351 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:53:29.351 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:53:29.351 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:53:29.351 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:29.351 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.343+0000 7f23b2fa5640 1 librados: starting msgr at 2024-03-29T11:53:29.351 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.343+0000 7f23b2fa5640 1 librados: starting objecter 2024-03-29T11:53:29.351 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.343+0000 7f23b2fa5640 1 Processor -- start 2024-03-29T11:53:29.351 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.343+0000 7f23b2fa5640 1 -- start start 2024-03-29T11:53:29.351 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.343+0000 7f23b2fa5640 1 librados: setting wanted keys 2024-03-29T11:53:29.352 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.343+0000 7f23b2fa5640 1 librados: calling monclient init 2024-03-29T11:53:29.352 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.343+0000 7f23b2fa5640 10 monclient: init 2024-03-29T11:53:29.352 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.344+0000 7f23b2fa5640 10 monclient: _reopen_session rank -1 2024-03-29T11:53:29.352 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.344+0000 7f23b2fa5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:53:29.352 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.345+0000 7f23b2fa5640 10 monclient(hunting): picked mon.noname-b con 0x7f23ac0a9930 addr v1:172.21.15.121:6789/0 2024-03-29T11:53:29.356 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.345+0000 7f23b2fa5640 10 monclient(hunting): picked mon.noname-c con 0x7f23ac149dc0 addr v1:172.21.15.98:6790/0 2024-03-29T11:53:29.356 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.345+0000 7f23b2fa5640 10 monclient(hunting): picked mon.noname-a con 0x7f23ac14ff40 addr v1:172.21.15.98:6789/0 2024-03-29T11:53:29.356 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.345+0000 7f23b2fa5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23ac1dbcf0 con 0x7f23ac14ff40 2024-03-29T11:53:29.356 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.345+0000 7f23b2fa5640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23ac1db9c0 con 0x7f23ac0a9930 2024-03-29T11:53:29.356 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.348+0000 7f23abfff640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f23ac149dc0 0x7f23ac0ac400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:52786/0 (socket says 172.21.15.98:52786) 2024-03-29T11:53:29.356 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.348+0000 7f23abfff640 1 -- 172.21.15.98:0/4045347224 learned_addr learned my addr 172.21.15.98:0/4045347224 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:53:29.356 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.348+0000 7f23b2fa5640 1 -- 172.21.15.98:0/4045347224 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23ac1dc890 con 0x7f23ac149dc0 2024-03-29T11:53:29.356 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.348+0000 7f23a97fa640 10 client.?.objecter ms_handle_connect 0x7f23ac149dc0 2024-03-29T11:53:29.357 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.348+0000 7f23a97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:53:29.357 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.348+0000 7f23b2fa5640 10 monclient(hunting): _renew_subs 2024-03-29T11:53:29.357 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.348+0000 7f23b2fa5640 10 monclient(hunting): authenticate will time out at 1116.433831s 2024-03-29T11:53:29.358 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.349+0000 7f23a97fa640 10 client.?.objecter ms_handle_connect 0x7f23ac14ff40 2024-03-29T11:53:29.358 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.349+0000 7f23a97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:53:29.358 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.349+0000 7f23a97fa640 10 client.?.objecter ms_handle_connect 0x7f23ac0a9930 2024-03-29T11:53:29.358 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.350+0000 7f23a97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:53:29.358 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.350+0000 7f23a97fa640 1 -- 172.21.15.98:0/4045347224 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1111227095 0 0) 0x7f23ac1dc890 con 0x7f23ac149dc0 2024-03-29T11:53:29.359 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.350+0000 7f23a97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:53:29.359 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.351+0000 7f23a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:29.359 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.351+0000 7f23a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:29.359 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.351+0000 7f23a97fa640 10 monclient(hunting): my global_id is 4430 2024-03-29T11:53:29.359 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.351+0000 7f23a97fa640 1 -- 172.21.15.98:0/4045347224 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f238c004eb0 con 0x7f23ac149dc0 2024-03-29T11:53:29.359 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.351+0000 7f23a97fa640 1 -- 172.21.15.98:0/4045347224 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 150796448 0 0) 0x7f23ac1db9c0 con 0x7f23ac0a9930 2024-03-29T11:53:29.362 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.351+0000 7f23a97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:53:29.362 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.351+0000 7f23a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:29.362 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.351+0000 7f23a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:29.362 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.351+0000 7f23a97fa640 10 monclient(hunting): my global_id is 4423 2024-03-29T11:53:29.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.351+0000 7f23a97fa640 1 -- 172.21.15.98:0/4045347224 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23ac1dc890 con 0x7f23ac0a9930 2024-03-29T11:53:29.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.351+0000 7f23a97fa640 1 -- 172.21.15.98:0/4045347224 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1923998670 0 0) 0x7f238c004eb0 con 0x7f23ac149dc0 2024-03-29T11:53:29.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.352+0000 7f23a97fa640 1 -- 172.21.15.98:0/4045347224 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23ac1db9c0 con 0x7f23ac149dc0 2024-03-29T11:53:29.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.352+0000 7f23a97fa640 1 -- 172.21.15.98:0/4045347224 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f239c001b70 con 0x7f23ac149dc0 2024-03-29T11:53:29.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.352+0000 7f23a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:29.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.352+0000 7f23a97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:53:29.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.352+0000 7f23a97fa640 10 monclient(hunting): dump: 2024-03-29T11:53:29.363 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:53:29.363 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:29.363 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:29.363 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:29.363 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:29.363 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:29.363 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:53:29.364 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:53:29.364 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:53:29.364 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:29.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.352+0000 7f23a97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:53:29.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.352+0000 7f23a97fa640 1 -- 172.21.15.98:0/4045347224 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 489596262 0 0) 0x7f23ac1dbcf0 con 0x7f23ac14ff40 2024-03-29T11:53:29.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.352+0000 7f23a97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:53:29.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.352+0000 7f23a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:29.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.352+0000 7f23a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:29.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.352+0000 7f23a97fa640 10 monclient(hunting): my global_id is 4395 2024-03-29T11:53:29.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.352+0000 7f23a97fa640 1 -- 172.21.15.98:0/4045347224 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f238c004eb0 con 0x7f23ac14ff40 2024-03-29T11:53:29.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.352+0000 7f23a97fa640 1 -- 172.21.15.98:0/4045347224 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2851545947 0 0) 0x7f23ac1db9c0 con 0x7f23ac149dc0 2024-03-29T11:53:29.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.352+0000 7f23a97fa640 1 -- 172.21.15.98:0/4045347224 >> v1:172.21.15.121:6789/0 conn(0x7f23ac0a9930 legacy=0x7f23ac1dc180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:29.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.352+0000 7f23a97fa640 1 -- 172.21.15.98:0/4045347224 >> v1:172.21.15.98:6789/0 conn(0x7f23ac14ff40 legacy=0x7f23ac146be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:29.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.352+0000 7f23a97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:53:29.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.352+0000 7f23a97fa640 1 monclient: found mon.c 2024-03-29T11:53:29.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.352+0000 7f23a97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:53:29.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.352+0000 7f23a97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:53:29.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.352+0000 7f23a97fa640 1 -- 172.21.15.98:0/4045347224 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23ac0acb10 con 0x7f23ac149dc0 2024-03-29T11:53:29.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.352+0000 7f23a97fa640 10 monclient: _finish_auth 0 2024-03-29T11:53:29.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.352+0000 7f23a97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:53:29.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.352+0000 7f23a97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:53:29.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.352+0000 7f23b2fa5640 5 monclient: authenticate success, global_id 4430 2024-03-29T11:53:29.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.352+0000 7f23b2fa5640 10 monclient: _renew_subs 2024-03-29T11:53:29.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.352+0000 7f23b2fa5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:53:29.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.352+0000 7f23b2fa5640 1 -- 172.21.15.98:0/4045347224 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23ac1472f0 con 0x7f23ac149dc0 2024-03-29T11:53:29.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.352+0000 7f23b2fa5640 10 client.4430.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:53:29.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.352+0000 7f23b2fa5640 10 monclient: _renew_subs 2024-03-29T11:53:29.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.352+0000 7f23b2fa5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:53:29.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.352+0000 7f23b2fa5640 1 -- 172.21.15.98:0/4045347224 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f23ac0abe20 con 0x7f23ac149dc0 2024-03-29T11:53:29.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.352+0000 7f23b2fa5640 1 librados: init done 2024-03-29T11:53:29.367 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.355+0000 7f23b2fa5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:53:29.367 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.355+0000 7f23b2fa5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:53:29.367 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.355+0000 7f23b2fa5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:53:29.367 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.355+0000 7f23b2fa5640 1 -- 172.21.15.98:0/4045347224 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23ac145130 con 0x7f23ac149dc0 2024-03-29T11:53:29.367 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.357+0000 7f23a97fa640 1 -- 172.21.15.98:0/4045347224 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f239c001cd0 con 0x7f23ac149dc0 2024-03-29T11:53:29.367 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.357+0000 7f23a97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:53:29.367 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.357+0000 7f23a97fa640 1 -- 172.21.15.98:0/4045347224 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f239c004d90 con 0x7f23ac149dc0 2024-03-29T11:53:29.367 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.357+0000 7f23a97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:29.367 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.357+0000 7f23a97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:53:29.367 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.357+0000 7f23a97fa640 10 monclient: dump: 2024-03-29T11:53:29.368 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:53:29.368 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:29.368 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:29.368 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:29.368 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:29.368 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:29.368 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:53:29.368 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:53:29.368 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:53:29.368 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:29.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.357+0000 7f23a97fa640 1 -- 172.21.15.98:0/4045347224 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f239c004fb0 con 0x7f23ac149dc0 2024-03-29T11:53:29.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.357+0000 7f23aaffd640 4 set_mon_vals no callback set 2024-03-29T11:53:29.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.358+0000 7f23b0d41640 10 monclient: get_auth_request con 0x7f238c06dd60 auth_method 0 2024-03-29T11:53:29.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.358+0000 7f23a97fa640 10 client.4430.objecter ms_handle_connect 0x7f238c06dd60 2024-03-29T11:53:29.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.358+0000 7f23a97fa640 1 -- 172.21.15.98:0/4045347224 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f239c084d30 con 0x7f23ac149dc0 2024-03-29T11:53:29.369 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.358+0000 7f23a97fa640 10 client.4430.objecter ms_dispatch 0x7f23ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:53:29.369 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.358+0000 7f23a97fa640 3 client.4430.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:53:29.369 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.358+0000 7f23a97fa640 3 client.4430.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:53:29.369 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.358+0000 7f23a97fa640 20 client.4430.objecter dump_active .. 0 homeless 2024-03-29T11:53:29.377 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.369+0000 7f23a97fa640 1 -- 172.21.15.98:0/4045347224 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f239c050690 con 0x7f23ac149dc0 2024-03-29T11:53:29.377 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.369+0000 7f23a97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:53:29.377 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.369+0000 7f23a97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:53:29.705 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.697+0000 7f23b2fa5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:53:29.705 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.697+0000 7f23b2fa5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:53:29.705 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.697+0000 7f23b2fa5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:53:29.705 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.697+0000 7f23b2fa5640 1 -- 172.21.15.98:0/4045347224 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23ac075f10 con 0x7f23ac149dc0 2024-03-29T11:53:29.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.699+0000 7f23a97fa640 1 -- 172.21.15.98:0/4045347224 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f239c0525c0 con 0x7f23ac149dc0 2024-03-29T11:53:29.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.699+0000 7f23a97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:53:29.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.699+0000 7f23a97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:53:29.707 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:53:29.707 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:53:29.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.703+0000 7f23b2fa5640 10 librados: watch_flush enter 2024-03-29T11:53:29.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.703+0000 7f23b2fa5640 10 librados: watch_flush exit 2024-03-29T11:53:29.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.703+0000 7f23b2fa5640 20 client.4430.objecter shutdown clearing up homeless session... 2024-03-29T11:53:29.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.703+0000 7f23b2fa5640 10 client.4430.objecter successfully canceled tick 2024-03-29T11:53:29.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.703+0000 7f23b2fa5640 1 -- 172.21.15.98:0/4045347224 >> v1:172.21.15.98:6816/4065186974 conn(0x7f238c06dd60 legacy=0x7f238c070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:29.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.703+0000 7f23b2fa5640 10 monclient: shutdown 2024-03-29T11:53:29.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.703+0000 7f23b2fa5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:53:29.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.703+0000 7f23b2fa5640 1 -- 172.21.15.98:0/4045347224 >> v1:172.21.15.98:6790/0 conn(0x7f23ac149dc0 legacy=0x7f23ac0ac400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:29.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.703+0000 7f23b2fa5640 1 -- 172.21.15.98:0/4045347224 shutdown_connections 2024-03-29T11:53:29.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.703+0000 7f23b2fa5640 1 -- 172.21.15.98:0/4045347224 >> 172.21.15.98:0/4045347224 conn(0x7f23ac0b35e0 msgr2=0x7f23ac0b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:53:29.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.704+0000 7f23b2fa5640 1 -- 172.21.15.98:0/4045347224 shutdown_connections 2024-03-29T11:53:29.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.704+0000 7f23b2fa5640 1 -- 172.21.15.98:0/4045347224 wait complete. 2024-03-29T11:53:29.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:29.704+0000 7f23b2fa5640 1 librados: shutdown 2024-03-29T11:53:29.730 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:53:36.537 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:53:36.684 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.677+0000 7f114a52d640 10 monclient: get_monmap_and_config 2024-03-29T11:53:36.684 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.677+0000 7f114a52d640 10 monclient: build_initial_monmap 2024-03-29T11:53:36.684 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.677+0000 7f114a52d640 10 monclient: monmap: 2024-03-29T11:53:36.685 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:53:36.685 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:36.685 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:53:36.677966+0000 2024-03-29T11:53:36.685 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:53:36.677966+0000 2024-03-29T11:53:36.685 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:53:36.685 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:36.685 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:53:36.685 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:53:36.685 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:53:36.685 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:36.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.678+0000 7f114a52d640 1 Processor -- start 2024-03-29T11:53:36.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.678+0000 7f114a52d640 1 -- start start 2024-03-29T11:53:36.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.678+0000 7f114a52d640 10 monclient: init 2024-03-29T11:53:36.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.679+0000 7f114a52d640 10 monclient: _reopen_session rank -1 2024-03-29T11:53:36.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.679+0000 7f114a52d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:53:36.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.679+0000 7f114a52d640 10 monclient(hunting): picked mon.noname-c con 0x7f1144070450 addr v1:172.21.15.121:6789/0 2024-03-29T11:53:36.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.679+0000 7f114a52d640 10 monclient(hunting): picked mon.noname-a con 0x7f11440b03b0 addr v1:172.21.15.98:6789/0 2024-03-29T11:53:36.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.679+0000 7f114a52d640 10 monclient(hunting): picked mon.noname-b con 0x7f114413d370 addr v1:172.21.15.98:6790/0 2024-03-29T11:53:36.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.679+0000 7f114a52d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f114413fdc0 con 0x7f11440b03b0 2024-03-29T11:53:36.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.679+0000 7f114a52d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1144071e40 con 0x7f1144070450 2024-03-29T11:53:36.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.679+0000 7f114a52d640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1144072120 con 0x7f114413d370 2024-03-29T11:53:36.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.679+0000 7f114a52d640 10 monclient(hunting): _renew_subs 2024-03-29T11:53:36.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.679+0000 7f114a52d640 10 monclient(hunting): authenticate will time out at 1123.764698s 2024-03-29T11:53:36.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.679+0000 7f1148d2a640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f11440b03b0 0x7f114413cc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38686/0 (socket says 172.21.15.98:38686) 2024-03-29T11:53:36.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.679+0000 7f1148d2a640 1 -- 172.21.15.98:0/2705876104 learned_addr learned my addr 172.21.15.98:0/2705876104 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:53:36.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.680+0000 7f113bfff640 1 -- 172.21.15.98:0/2705876104 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1837890245 0 0) 0x7f114413fdc0 con 0x7f11440b03b0 2024-03-29T11:53:36.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.680+0000 7f113bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:53:36.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.680+0000 7f113bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:36.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.680+0000 7f113bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:36.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.680+0000 7f113bfff640 10 monclient(hunting): my global_id is 4398 2024-03-29T11:53:36.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.680+0000 7f113bfff640 1 -- 172.21.15.98:0/2705876104 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1130003520 con 0x7f11440b03b0 2024-03-29T11:53:36.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.680+0000 7f113bfff640 1 -- 172.21.15.98:0/2705876104 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3056350205 0 0) 0x7f1130003520 con 0x7f11440b03b0 2024-03-29T11:53:36.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.680+0000 7f113bfff640 1 -- 172.21.15.98:0/2705876104 >> v1:172.21.15.98:6790/0 conn(0x7f114413d370 legacy=0x7f114414b520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:53:36.690 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.682+0000 7f113bfff640 1 -- 172.21.15.98:0/2705876104 >> v1:172.21.15.121:6789/0 conn(0x7f1144070450 legacy=0x7f1144140250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:53:36.692 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.684+0000 7f113bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:53:36.692 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.684+0000 7f113bfff640 1 monclient: found mon.noname-a 2024-03-29T11:53:36.692 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.684+0000 7f113bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:53:36.692 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.684+0000 7f113bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T11:53:36.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.684+0000 7f113bfff640 1 -- 172.21.15.98:0/2705876104 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1144140960 con 0x7f11440b03b0 2024-03-29T11:53:36.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.684+0000 7f113bfff640 10 monclient: _finish_auth 0 2024-03-29T11:53:36.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.684+0000 7f113bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:53:36.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.684+0000 7f113bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:53:36.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.684+0000 7f114a52d640 5 monclient: authenticate success, global_id 4398 2024-03-29T11:53:36.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.684+0000 7f114a52d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:53:36.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.684+0000 7f113bfff640 1 -- 172.21.15.98:0/2705876104 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f1134001690 con 0x7f11440b03b0 2024-03-29T11:53:36.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.684+0000 7f113bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:36.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.684+0000 7f113bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:53:36.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.684+0000 7f113bfff640 10 monclient: dump: 2024-03-29T11:53:36.693 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:53:36.694 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:36.694 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:36.694 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:36.694 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:36.694 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:36.694 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:53:36.694 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:53:36.694 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:53:36.694 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:36.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.684+0000 7f114a52d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:53:36.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.684+0000 7f113bfff640 1 -- 172.21.15.98:0/2705876104 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1134001920 con 0x7f11440b03b0 2024-03-29T11:53:36.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.684+0000 7f113bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:53:36.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.684+0000 7f114a52d640 10 monclient: get_monmap_and_config success 2024-03-29T11:53:36.695 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.684+0000 7f114a52d640 4 set_mon_vals no callback set 2024-03-29T11:53:36.695 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.685+0000 7f113bfff640 1 -- 172.21.15.98:0/2705876104 <== mon.0 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f1134001cb0 con 0x7f11440b03b0 2024-03-29T11:53:36.695 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.685+0000 7f113bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:36.695 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.685+0000 7f113bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:53:36.695 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.685+0000 7f113bfff640 10 monclient: dump: 2024-03-29T11:53:36.695 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:53:36.695 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:36.695 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:36.695 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:36.695 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:36.695 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:36.695 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:53:36.695 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:53:36.695 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:53:36.695 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:36.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.685+0000 7f114a52d640 10 monclient: shutdown 2024-03-29T11:53:36.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.685+0000 7f114a52d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:53:36.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.685+0000 7f114a52d640 1 -- 172.21.15.98:0/2705876104 >> v1:172.21.15.98:6789/0 conn(0x7f11440b03b0 legacy=0x7f114413cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:36.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.685+0000 7f114a52d640 1 -- 172.21.15.98:0/2705876104 shutdown_connections 2024-03-29T11:53:36.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.685+0000 7f114a52d640 1 -- 172.21.15.98:0/2705876104 >> 172.21.15.98:0/2705876104 conn(0x7f11440671c0 msgr2=0x7f11440b59b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:53:36.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.687+0000 7f114a52d640 1 -- 172.21.15.98:0/2705876104 shutdown_connections 2024-03-29T11:53:36.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.687+0000 7f114a52d640 1 -- 172.21.15.98:0/2705876104 wait complete. 2024-03-29T11:53:36.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.688+0000 7f114a52d640 10 monclient: build_initial_monmap 2024-03-29T11:53:36.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.688+0000 7f114a52d640 10 monclient: monmap: 2024-03-29T11:53:36.699 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:53:36.699 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:53:36.699 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:53:36.699 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:53:36.699 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:53:36.699 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:36.699 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:53:36.699 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:53:36.699 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:53:36.699 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:36.699 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.688+0000 7f114a52d640 1 librados: starting msgr at 2024-03-29T11:53:36.699 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.688+0000 7f114a52d640 1 librados: starting objecter 2024-03-29T11:53:36.699 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.688+0000 7f114a52d640 1 Processor -- start 2024-03-29T11:53:36.699 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.689+0000 7f114a52d640 1 -- start start 2024-03-29T11:53:36.699 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.689+0000 7f114a52d640 1 librados: setting wanted keys 2024-03-29T11:53:36.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.689+0000 7f114a52d640 1 librados: calling monclient init 2024-03-29T11:53:36.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.689+0000 7f114a52d640 10 monclient: init 2024-03-29T11:53:36.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.689+0000 7f114a52d640 10 monclient: _reopen_session rank -1 2024-03-29T11:53:36.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.689+0000 7f114a52d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:53:36.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.689+0000 7f114a52d640 10 monclient(hunting): picked mon.noname-b con 0x7f1144070450 addr v1:172.21.15.121:6789/0 2024-03-29T11:53:36.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.689+0000 7f114a52d640 10 monclient(hunting): picked mon.noname-c con 0x7f11440b03b0 addr v1:172.21.15.98:6790/0 2024-03-29T11:53:36.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.689+0000 7f114a52d640 10 monclient(hunting): picked mon.noname-a con 0x7f114413d370 addr v1:172.21.15.98:6789/0 2024-03-29T11:53:36.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.689+0000 7f114a52d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11441351b0 con 0x7f114413d370 2024-03-29T11:53:36.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.689+0000 7f114a52d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1144138600 con 0x7f1144070450 2024-03-29T11:53:36.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.689+0000 7f114a52d640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11441388e0 con 0x7f11440b03b0 2024-03-29T11:53:36.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.689+0000 7f114a52d640 10 monclient(hunting): _renew_subs 2024-03-29T11:53:36.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.689+0000 7f114a52d640 10 monclient(hunting): authenticate will time out at 1123.774944s 2024-03-29T11:53:36.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.693+0000 7f1148d2a640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f11440b03b0 0x7f114406bc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:54365/0 (socket says 172.21.15.98:54365) 2024-03-29T11:53:36.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.693+0000 7f1148d2a640 1 -- 172.21.15.98:0/152691877 learned_addr learned my addr 172.21.15.98:0/152691877 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:53:36.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.693+0000 7f1139ffb640 10 client.?.objecter ms_handle_connect 0x7f11440b03b0 2024-03-29T11:53:36.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.693+0000 7f1139ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:53:36.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.693+0000 7f1139ffb640 1 -- 172.21.15.98:0/152691877 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1011454323 0 0) 0x7f11441388e0 con 0x7f11440b03b0 2024-03-29T11:53:36.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.693+0000 7f1139ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:53:36.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.693+0000 7f1139ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:36.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.693+0000 7f1139ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:36.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.693+0000 7f1139ffb640 10 monclient(hunting): my global_id is 4433 2024-03-29T11:53:36.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.693+0000 7f1139ffb640 1 -- 172.21.15.98:0/152691877 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1128004eb0 con 0x7f11440b03b0 2024-03-29T11:53:36.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.693+0000 7f1139ffb640 10 client.?.objecter ms_handle_connect 0x7f114413d370 2024-03-29T11:53:36.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.693+0000 7f1139ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:53:36.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.693+0000 7f1139ffb640 10 client.?.objecter ms_handle_connect 0x7f1144070450 2024-03-29T11:53:36.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.693+0000 7f1139ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:53:36.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.693+0000 7f1139ffb640 1 -- 172.21.15.98:0/152691877 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4215022974 0 0) 0x7f1144138600 con 0x7f1144070450 2024-03-29T11:53:36.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.693+0000 7f1139ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:53:36.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.693+0000 7f1139ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:36.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.693+0000 7f1139ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:36.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.693+0000 7f1139ffb640 10 monclient(hunting): my global_id is 4426 2024-03-29T11:53:36.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.693+0000 7f1139ffb640 1 -- 172.21.15.98:0/152691877 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11441388e0 con 0x7f1144070450 2024-03-29T11:53:36.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.693+0000 7f1139ffb640 1 -- 172.21.15.98:0/152691877 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3579364718 0 0) 0x7f11441351b0 con 0x7f114413d370 2024-03-29T11:53:36.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.693+0000 7f1139ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:53:36.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.693+0000 7f1139ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:36.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.693+0000 7f1139ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:36.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.693+0000 7f1139ffb640 10 monclient(hunting): my global_id is 4401 2024-03-29T11:53:36.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.693+0000 7f1139ffb640 1 -- 172.21.15.98:0/152691877 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1144138600 con 0x7f114413d370 2024-03-29T11:53:36.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.694+0000 7f1139ffb640 1 -- 172.21.15.98:0/152691877 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4030185547 0 0) 0x7f1128004eb0 con 0x7f11440b03b0 2024-03-29T11:53:36.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.694+0000 7f1139ffb640 1 -- 172.21.15.98:0/152691877 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11441351b0 con 0x7f11440b03b0 2024-03-29T11:53:36.709 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.694+0000 7f1139ffb640 1 -- 172.21.15.98:0/152691877 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f1134001610 con 0x7f11440b03b0 2024-03-29T11:53:36.709 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.694+0000 7f1139ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:36.709 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.694+0000 7f1139ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:53:36.709 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.694+0000 7f1139ffb640 10 monclient(hunting): dump: 2024-03-29T11:53:36.709 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:53:36.709 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:36.709 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:36.709 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:36.709 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:36.709 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:36.709 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:53:36.709 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:53:36.709 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:53:36.709 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:36.710 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.694+0000 7f1139ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:53:36.710 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.694+0000 7f1139ffb640 1 -- 172.21.15.98:0/152691877 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 6293385 0 0) 0x7f11441351b0 con 0x7f11440b03b0 2024-03-29T11:53:36.710 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.694+0000 7f1139ffb640 1 -- 172.21.15.98:0/152691877 >> v1:172.21.15.121:6789/0 conn(0x7f1144070450 legacy=0x7f1144139350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:36.710 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.694+0000 7f1139ffb640 1 -- 172.21.15.98:0/152691877 >> v1:172.21.15.98:6789/0 conn(0x7f114413d370 legacy=0x7f11441e5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:36.710 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.694+0000 7f1139ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:53:36.710 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.694+0000 7f1139ffb640 1 monclient: found mon.c 2024-03-29T11:53:36.710 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.694+0000 7f1139ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:53:36.710 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.694+0000 7f1139ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:53:36.710 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.694+0000 7f1139ffb640 1 -- 172.21.15.98:0/152691877 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1144137000 con 0x7f11440b03b0 2024-03-29T11:53:36.710 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.694+0000 7f1139ffb640 10 monclient: _finish_auth 0 2024-03-29T11:53:36.710 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.694+0000 7f1139ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:53:36.710 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.694+0000 7f1139ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:53:36.710 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.694+0000 7f114a52d640 5 monclient: authenticate success, global_id 4433 2024-03-29T11:53:36.710 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.694+0000 7f114a52d640 10 monclient: _renew_subs 2024-03-29T11:53:36.710 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.694+0000 7f114a52d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:53:36.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.694+0000 7f114a52d640 1 -- 172.21.15.98:0/152691877 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11441379a0 con 0x7f11440b03b0 2024-03-29T11:53:36.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.695+0000 7f114a52d640 10 client.4433.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:53:36.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.695+0000 7f114a52d640 10 monclient: _renew_subs 2024-03-29T11:53:36.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.695+0000 7f114a52d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:53:36.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.695+0000 7f114a52d640 1 -- 172.21.15.98:0/152691877 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1144139c40 con 0x7f11440b03b0 2024-03-29T11:53:36.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.695+0000 7f114a52d640 1 librados: init done 2024-03-29T11:53:36.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.695+0000 7f1139ffb640 1 -- 172.21.15.98:0/152691877 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1134004bf0 con 0x7f11440b03b0 2024-03-29T11:53:36.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.695+0000 7f1139ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:53:36.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.695+0000 7f113b7fe640 4 set_mon_vals no callback set 2024-03-29T11:53:36.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.696+0000 7f1139ffb640 1 -- 172.21.15.98:0/152691877 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f1134004f00 con 0x7f11440b03b0 2024-03-29T11:53:36.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.696+0000 7f1139ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:36.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.696+0000 7f1139ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:53:36.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.696+0000 7f1139ffb640 10 monclient: dump: 2024-03-29T11:53:36.711 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:53:36.712 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:36.712 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:36.712 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:36.712 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:36.712 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:36.712 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:53:36.712 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:53:36.712 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:53:36.712 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:36.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.696+0000 7f114a52d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:53:36.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.696+0000 7f114a52d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:53:36.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.696+0000 7f114a52d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:53:36.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.696+0000 7f114a52d640 1 -- 172.21.15.98:0/152691877 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1124005080 con 0x7f11440b03b0 2024-03-29T11:53:36.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.697+0000 7f1139ffb640 1 -- 172.21.15.98:0/152691877 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f113401aed0 con 0x7f11440b03b0 2024-03-29T11:53:36.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.698+0000 7f1139ffb640 1 -- 172.21.15.98:0/152691877 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f1134084bf0 con 0x7f11440b03b0 2024-03-29T11:53:36.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.698+0000 7f1139ffb640 10 client.4433.objecter ms_dispatch 0x7f1144000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:53:36.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.698+0000 7f1139ffb640 3 client.4433.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:53:36.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.698+0000 7f1139ffb640 3 client.4433.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:53:36.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.698+0000 7f1139ffb640 20 client.4433.objecter dump_active .. 0 homeless 2024-03-29T11:53:36.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.702+0000 7f114952b640 10 monclient: get_auth_request con 0x7f112806dea0 auth_method 0 2024-03-29T11:53:36.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.703+0000 7f1139ffb640 10 client.4433.objecter ms_handle_connect 0x7f112806dea0 2024-03-29T11:53:36.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.707+0000 7f1139ffb640 1 -- 172.21.15.98:0/152691877 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f11340504d0 con 0x7f11440b03b0 2024-03-29T11:53:36.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.707+0000 7f1139ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:53:36.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:36.707+0000 7f1139ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:53:37.038 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:37.030+0000 7f114a52d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:53:37.038 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:37.030+0000 7f114a52d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:53:37.038 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:37.030+0000 7f114a52d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:53:37.038 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:37.030+0000 7f114a52d640 1 -- 172.21.15.98:0/152691877 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1124005330 con 0x7f11440b03b0 2024-03-29T11:53:37.039 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:37.032+0000 7f1139ffb640 1 -- 172.21.15.98:0/152691877 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f1134052400 con 0x7f11440b03b0 2024-03-29T11:53:37.039 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:37.032+0000 7f1139ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:53:37.039 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:37.032+0000 7f1139ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:53:37.040 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:53:37.040 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:53:37.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:37.035+0000 7f114a52d640 10 librados: watch_flush enter 2024-03-29T11:53:37.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:37.035+0000 7f114a52d640 10 librados: watch_flush exit 2024-03-29T11:53:37.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:37.035+0000 7f114a52d640 20 client.4433.objecter shutdown clearing up homeless session... 2024-03-29T11:53:37.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:37.035+0000 7f114a52d640 10 client.4433.objecter successfully canceled tick 2024-03-29T11:53:37.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:37.036+0000 7f114a52d640 1 -- 172.21.15.98:0/152691877 >> v1:172.21.15.98:6816/4065186974 conn(0x7f112806dea0 legacy=0x7f1128070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:37.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:37.036+0000 7f114a52d640 10 monclient: shutdown 2024-03-29T11:53:37.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:37.036+0000 7f114a52d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:53:37.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:37.036+0000 7f114a52d640 1 -- 172.21.15.98:0/152691877 >> v1:172.21.15.98:6790/0 conn(0x7f11440b03b0 legacy=0x7f114406bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:37.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:37.037+0000 7f114a52d640 1 -- 172.21.15.98:0/152691877 shutdown_connections 2024-03-29T11:53:37.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:37.037+0000 7f114a52d640 1 -- 172.21.15.98:0/152691877 >> 172.21.15.98:0/152691877 conn(0x7f11440671c0 msgr2=0x7f114413ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:53:37.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:37.037+0000 7f114a52d640 1 -- 172.21.15.98:0/152691877 shutdown_connections 2024-03-29T11:53:37.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:37.037+0000 7f114a52d640 1 -- 172.21.15.98:0/152691877 wait complete. 2024-03-29T11:53:37.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:37.037+0000 7f114a52d640 1 librados: shutdown 2024-03-29T11:53:37.061 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:53:43.867 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:53:44.036 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.028+0000 7f4149693640 10 monclient: get_monmap_and_config 2024-03-29T11:53:44.036 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.028+0000 7f4149693640 10 monclient: build_initial_monmap 2024-03-29T11:53:44.036 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.028+0000 7f4149693640 10 monclient: monmap: 2024-03-29T11:53:44.036 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:53:44.036 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:44.036 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:53:44.029443+0000 2024-03-29T11:53:44.036 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:53:44.029443+0000 2024-03-29T11:53:44.036 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:53:44.036 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:44.036 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:53:44.036 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:53:44.036 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:53:44.037 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:44.038 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.031+0000 7f4149693640 1 Processor -- start 2024-03-29T11:53:44.038 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.031+0000 7f4149693640 1 -- start start 2024-03-29T11:53:44.038 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.031+0000 7f4149693640 10 monclient: init 2024-03-29T11:53:44.040 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.032+0000 7f4149693640 10 monclient: _reopen_session rank -1 2024-03-29T11:53:44.040 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.032+0000 7f4149693640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:53:44.040 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.032+0000 7f4149693640 10 monclient(hunting): picked mon.noname-c con 0x7f41440a9d60 addr v1:172.21.15.121:6789/0 2024-03-29T11:53:44.040 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.032+0000 7f4149693640 10 monclient(hunting): picked mon.noname-b con 0x7f41440aa140 addr v1:172.21.15.98:6790/0 2024-03-29T11:53:44.041 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.034+0000 7f4142ffd640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f41440a9d60 0x7f41440ae650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:33068/0 (socket says 172.21.15.98:33068) 2024-03-29T11:53:44.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.035+0000 7f4149693640 10 monclient(hunting): picked mon.noname-a con 0x7f414413bad0 addr v1:172.21.15.98:6789/0 2024-03-29T11:53:44.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.035+0000 7f4142ffd640 1 -- 172.21.15.98:0/423025451 learned_addr learned my addr 172.21.15.98:0/423025451 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:53:44.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.035+0000 7f4149693640 1 -- 172.21.15.98:0/423025451 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41440ac290 con 0x7f414413bad0 2024-03-29T11:53:44.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.035+0000 7f4149693640 1 -- 172.21.15.98:0/423025451 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41440ae2e0 con 0x7f41440a9d60 2024-03-29T11:53:44.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.035+0000 7f4149693640 1 -- 172.21.15.98:0/423025451 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4144141b40 con 0x7f41440aa140 2024-03-29T11:53:44.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.035+0000 7f4149693640 10 monclient(hunting): _renew_subs 2024-03-29T11:53:44.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.035+0000 7f4149693640 10 monclient(hunting): authenticate will time out at 1131.120595s 2024-03-29T11:53:44.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.035+0000 7f41427fc640 1 -- 172.21.15.98:0/423025451 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1516604726 0 0) 0x7f41440ae2e0 con 0x7f41440a9d60 2024-03-29T11:53:44.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.035+0000 7f41427fc640 10 monclient(hunting): _negotiate 2024-03-29T11:53:44.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.035+0000 7f41427fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:44.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.035+0000 7f41427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:44.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.035+0000 7f41427fc640 10 monclient(hunting): my global_id is 4429 2024-03-29T11:53:44.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.036+0000 7f41427fc640 1 -- 172.21.15.98:0/423025451 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4130003520 con 0x7f41440a9d60 2024-03-29T11:53:44.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.036+0000 7f41427fc640 1 -- 172.21.15.98:0/423025451 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 920681799 0 0) 0x7f4130003520 con 0x7f41440a9d60 2024-03-29T11:53:44.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.036+0000 7f41427fc640 1 -- 172.21.15.98:0/423025451 >> v1:172.21.15.98:6790/0 conn(0x7f41440aa140 legacy=0x7f414413b3c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:53:44.047 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.038+0000 7f41427fc640 1 -- 172.21.15.98:0/423025451 >> v1:172.21.15.98:6789/0 conn(0x7f414413bad0 legacy=0x7f4144149a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:44.047 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.038+0000 7f41427fc640 10 monclient: _finish_hunting 0 2024-03-29T11:53:44.047 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.038+0000 7f41427fc640 1 monclient: found mon.noname-c 2024-03-29T11:53:44.047 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.039+0000 7f41427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:53:44.047 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.039+0000 7f41427fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:53:44.047 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.039+0000 7f41427fc640 1 -- 172.21.15.98:0/423025451 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41440aaf50 con 0x7f41440a9d60 2024-03-29T11:53:44.047 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.039+0000 7f41427fc640 10 monclient: _finish_auth 0 2024-03-29T11:53:44.048 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.039+0000 7f41427fc640 10 monclient: _check_auth_tickets 2024-03-29T11:53:44.048 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.039+0000 7f41427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:53:44.048 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.039+0000 7f41427fc640 1 -- 172.21.15.98:0/423025451 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f4134001690 con 0x7f41440a9d60 2024-03-29T11:53:44.048 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.039+0000 7f41427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:44.048 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.039+0000 7f41427fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:53:44.048 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.039+0000 7f41427fc640 10 monclient: dump: 2024-03-29T11:53:44.048 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:53:44.048 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:44.048 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:44.048 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:44.048 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:44.048 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:44.048 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:53:44.048 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:53:44.048 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:53:44.049 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:44.049 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.039+0000 7f41427fc640 1 -- 172.21.15.98:0/423025451 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4134001860 con 0x7f41440a9d60 2024-03-29T11:53:44.049 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.039+0000 7f41427fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:53:44.049 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.039+0000 7f41427fc640 1 -- 172.21.15.98:0/423025451 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f4134001cb0 con 0x7f41440a9d60 2024-03-29T11:53:44.049 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.039+0000 7f41427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:44.049 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.039+0000 7f41427fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:53:44.049 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.039+0000 7f41427fc640 10 monclient: dump: 2024-03-29T11:53:44.049 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:53:44.049 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:44.049 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:44.049 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:44.049 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:44.049 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:44.050 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:53:44.050 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:53:44.050 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:53:44.050 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:44.050 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.040+0000 7f4149693640 5 monclient: authenticate success, global_id 4429 2024-03-29T11:53:44.050 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.040+0000 7f4149693640 10 monclient: get_monmap_and_config success 2024-03-29T11:53:44.050 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.040+0000 7f4149693640 4 set_mon_vals no callback set 2024-03-29T11:53:44.051 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.041+0000 7f4149693640 10 monclient: shutdown 2024-03-29T11:53:44.051 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.041+0000 7f4149693640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:53:44.051 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.041+0000 7f4149693640 1 -- 172.21.15.98:0/423025451 >> v1:172.21.15.121:6789/0 conn(0x7f41440a9d60 legacy=0x7f41440ae650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:44.051 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.041+0000 7f4149693640 1 -- 172.21.15.98:0/423025451 shutdown_connections 2024-03-29T11:53:44.051 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.041+0000 7f4149693640 1 -- 172.21.15.98:0/423025451 >> 172.21.15.98:0/423025451 conn(0x7f41440b3c50 msgr2=0x7f41440b4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:53:44.051 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.042+0000 7f4149693640 1 -- 172.21.15.98:0/423025451 shutdown_connections 2024-03-29T11:53:44.051 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.043+0000 7f4149693640 1 -- 172.21.15.98:0/423025451 wait complete. 2024-03-29T11:53:44.053 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.044+0000 7f4149693640 10 monclient: build_initial_monmap 2024-03-29T11:53:44.053 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.044+0000 7f4149693640 10 monclient: monmap: 2024-03-29T11:53:44.053 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:53:44.054 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:53:44.054 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:53:44.054 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:53:44.054 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:53:44.054 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:44.054 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:53:44.054 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:53:44.054 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:53:44.054 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:44.054 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.044+0000 7f4149693640 1 librados: starting msgr at 2024-03-29T11:53:44.054 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.044+0000 7f4149693640 1 librados: starting objecter 2024-03-29T11:53:44.054 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.044+0000 7f4149693640 1 Processor -- start 2024-03-29T11:53:44.055 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.046+0000 7f4149693640 1 -- start start 2024-03-29T11:53:44.055 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.046+0000 7f4149693640 1 librados: setting wanted keys 2024-03-29T11:53:44.055 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.047+0000 7f4149693640 1 librados: calling monclient init 2024-03-29T11:53:44.056 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.047+0000 7f4149693640 10 monclient: init 2024-03-29T11:53:44.056 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.048+0000 7f4149693640 10 monclient: _reopen_session rank -1 2024-03-29T11:53:44.056 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.048+0000 7f4149693640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:53:44.060 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.048+0000 7f4149693640 10 monclient(hunting): picked mon.noname-a con 0x7f41440a9d60 addr v1:172.21.15.98:6789/0 2024-03-29T11:53:44.060 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.052+0000 7f4142ffd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f41440a9d60 0x7f4144135db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56563/0 (socket says 172.21.15.98:56563) 2024-03-29T11:53:44.060 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.052+0000 7f4142ffd640 1 -- 172.21.15.98:0/808924995 learned_addr learned my addr 172.21.15.98:0/808924995 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:53:44.060 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.052+0000 7f4149693640 10 monclient(hunting): picked mon.noname-c con 0x7f41440aa140 addr v1:172.21.15.98:6790/0 2024-03-29T11:53:44.060 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.052+0000 7f4149693640 10 monclient(hunting): picked mon.noname-b con 0x7f414413bad0 addr v1:172.21.15.121:6789/0 2024-03-29T11:53:44.060 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.052+0000 7f4149693640 1 -- 172.21.15.98:0/808924995 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4144070e90 con 0x7f41440a9d60 2024-03-29T11:53:44.060 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.052+0000 7f413b7fe640 10 client.?.objecter ms_handle_connect 0x7f41440a9d60 2024-03-29T11:53:44.060 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.052+0000 7f413b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:53:44.065 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.052+0000 7f4149693640 1 -- 172.21.15.98:0/808924995 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4144141240 con 0x7f414413bad0 2024-03-29T11:53:44.065 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.052+0000 7f4149693640 1 -- 172.21.15.98:0/808924995 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41441413d0 con 0x7f41440aa140 2024-03-29T11:53:44.065 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.052+0000 7f4149693640 10 monclient(hunting): _renew_subs 2024-03-29T11:53:44.065 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.052+0000 7f4149693640 10 monclient(hunting): authenticate will time out at 1131.137900s 2024-03-29T11:53:44.065 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.053+0000 7f413b7fe640 10 client.?.objecter ms_handle_connect 0x7f414413bad0 2024-03-29T11:53:44.066 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.053+0000 7f413b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:53:44.066 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.053+0000 7f413b7fe640 1 -- 172.21.15.98:0/808924995 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3122677196 0 0) 0x7f4144141240 con 0x7f414413bad0 2024-03-29T11:53:44.066 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.053+0000 7f413b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:53:44.066 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.053+0000 7f413b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:44.066 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.053+0000 7f413b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:44.066 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.053+0000 7f413b7fe640 10 monclient(hunting): my global_id is 4432 2024-03-29T11:53:44.066 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.053+0000 7f413b7fe640 1 -- 172.21.15.98:0/808924995 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4124004eb0 con 0x7f414413bad0 2024-03-29T11:53:44.066 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.053+0000 7f413b7fe640 1 -- 172.21.15.98:0/808924995 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1571901483 0 0) 0x7f4144070e90 con 0x7f41440a9d60 2024-03-29T11:53:44.066 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.053+0000 7f413b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:53:44.066 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.053+0000 7f413b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:44.066 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.053+0000 7f413b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:44.066 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.053+0000 7f413b7fe640 10 monclient(hunting): my global_id is 4407 2024-03-29T11:53:44.066 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.053+0000 7f413b7fe640 1 -- 172.21.15.98:0/808924995 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4144141240 con 0x7f41440a9d60 2024-03-29T11:53:44.067 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.053+0000 7f413b7fe640 1 -- 172.21.15.98:0/808924995 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3022417749 0 0) 0x7f4144141240 con 0x7f41440a9d60 2024-03-29T11:53:44.067 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.054+0000 7f413b7fe640 1 -- 172.21.15.98:0/808924995 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4144070e90 con 0x7f41440a9d60 2024-03-29T11:53:44.067 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.054+0000 7f413b7fe640 1 -- 172.21.15.98:0/808924995 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f4134001b70 con 0x7f41440a9d60 2024-03-29T11:53:44.067 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.054+0000 7f413b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:44.067 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.054+0000 7f413b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:53:44.067 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.054+0000 7f413b7fe640 10 monclient(hunting): dump: 2024-03-29T11:53:44.067 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:53:44.067 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:44.067 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:44.067 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:44.067 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:44.067 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:44.067 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:53:44.067 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:53:44.068 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:53:44.068 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:44.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.054+0000 7f413b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:53:44.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.054+0000 7f413b7fe640 1 -- 172.21.15.98:0/808924995 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1455337952 0 0) 0x7f4124004eb0 con 0x7f414413bad0 2024-03-29T11:53:44.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.054+0000 7f413b7fe640 1 -- 172.21.15.98:0/808924995 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4144141240 con 0x7f414413bad0 2024-03-29T11:53:44.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.055+0000 7f413b7fe640 1 -- 172.21.15.98:0/808924995 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f413c001a60 con 0x7f414413bad0 2024-03-29T11:53:44.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.055+0000 7f413b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:44.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.055+0000 7f413b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:53:44.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.055+0000 7f413b7fe640 10 monclient(hunting): dump: 2024-03-29T11:53:44.068 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:53:44.068 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:44.068 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:44.068 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:44.068 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:44.068 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:44.069 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:53:44.069 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:53:44.069 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:53:44.069 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:44.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.055+0000 7f413b7fe640 10 client.?.objecter ms_handle_connect 0x7f41440aa140 2024-03-29T11:53:44.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.055+0000 7f413b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:53:44.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.055+0000 7f413b7fe640 1 -- 172.21.15.98:0/808924995 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1187787980 0 0) 0x7f4144141240 con 0x7f414413bad0 2024-03-29T11:53:44.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.055+0000 7f413b7fe640 1 -- 172.21.15.98:0/808924995 >> v1:172.21.15.98:6790/0 conn(0x7f41440aa140 legacy=0x7f41441d6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:44.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.055+0000 7f413b7fe640 1 -- 172.21.15.98:0/808924995 >> v1:172.21.15.98:6789/0 conn(0x7f41440a9d60 legacy=0x7f4144135db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:44.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.055+0000 7f413b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:53:44.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.055+0000 7f413b7fe640 1 monclient: found mon.b 2024-03-29T11:53:44.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.055+0000 7f413b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:53:44.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.055+0000 7f413b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:53:44.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.055+0000 7f413b7fe640 1 -- 172.21.15.98:0/808924995 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41441364c0 con 0x7f414413bad0 2024-03-29T11:53:44.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.056+0000 7f413b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:53:44.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.056+0000 7f413b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:53:44.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.056+0000 7f413b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:53:44.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.056+0000 7f4149693640 5 monclient: authenticate success, global_id 4432 2024-03-29T11:53:44.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.056+0000 7f4149693640 10 monclient: _renew_subs 2024-03-29T11:53:44.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.056+0000 7f4149693640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:53:44.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.056+0000 7f4149693640 1 -- 172.21.15.98:0/808924995 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4144136710 con 0x7f414413bad0 2024-03-29T11:53:44.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.056+0000 7f4149693640 10 client.4432.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:53:44.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.056+0000 7f4149693640 10 monclient: _renew_subs 2024-03-29T11:53:44.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.056+0000 7f4149693640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:53:44.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.056+0000 7f4149693640 1 -- 172.21.15.98:0/808924995 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f41441d6d20 con 0x7f414413bad0 2024-03-29T11:53:44.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.056+0000 7f4149693640 1 librados: init done 2024-03-29T11:53:44.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.056+0000 7f413b7fe640 1 -- 172.21.15.98:0/808924995 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f413c001bc0 con 0x7f414413bad0 2024-03-29T11:53:44.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.056+0000 7f413b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:53:44.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.056+0000 7f413b7fe640 1 -- 172.21.15.98:0/808924995 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f413c003b10 con 0x7f414413bad0 2024-03-29T11:53:44.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.057+0000 7f4141ffb640 4 set_mon_vals no callback set 2024-03-29T11:53:44.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.057+0000 7f413b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:44.072 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.058+0000 7f413b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:53:44.072 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.058+0000 7f413b7fe640 10 monclient: dump: 2024-03-29T11:53:44.072 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:53:44.072 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:44.072 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:44.072 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:44.072 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:44.072 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:44.073 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:53:44.073 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:53:44.073 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:53:44.073 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:44.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.059+0000 7f4149693640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:53:44.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.060+0000 7f4149693640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:53:44.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.060+0000 7f4149693640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:53:44.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.060+0000 7f4149693640 1 -- 172.21.15.98:0/808924995 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4128005080 con 0x7f414413bad0 2024-03-29T11:53:44.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.060+0000 7f413b7fe640 1 -- 172.21.15.98:0/808924995 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f413c019ae0 con 0x7f414413bad0 2024-03-29T11:53:44.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.061+0000 7f413b7fe640 1 -- 172.21.15.98:0/808924995 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f413c083990 con 0x7f414413bad0 2024-03-29T11:53:44.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.061+0000 7f413b7fe640 10 client.4432.objecter ms_dispatch 0x7f4144000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:53:44.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.061+0000 7f413b7fe640 3 client.4432.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:53:44.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.061+0000 7f413b7fe640 3 client.4432.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:53:44.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.061+0000 7f413b7fe640 20 client.4432.objecter dump_active .. 0 homeless 2024-03-29T11:53:44.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.061+0000 7f4142ffd640 10 monclient: get_auth_request con 0x7f412406de10 auth_method 0 2024-03-29T11:53:44.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.062+0000 7f413b7fe640 10 client.4432.objecter ms_handle_connect 0x7f412406de10 2024-03-29T11:53:44.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.067+0000 7f413b7fe640 1 -- 172.21.15.98:0/808924995 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f413c04f270 con 0x7f414413bad0 2024-03-29T11:53:44.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.067+0000 7f413b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:53:44.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.067+0000 7f413b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:53:44.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.414+0000 7f4149693640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:53:44.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.414+0000 7f4149693640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:53:44.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.414+0000 7f4149693640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:53:44.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.414+0000 7f4149693640 1 -- 172.21.15.98:0/808924995 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4128005330 con 0x7f414413bad0 2024-03-29T11:53:44.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.415+0000 7f413b7fe640 1 -- 172.21.15.98:0/808924995 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f413c0511a0 con 0x7f414413bad0 2024-03-29T11:53:44.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.415+0000 7f413b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:53:44.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.415+0000 7f413b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:53:44.424 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:53:44.424 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:53:44.429 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.421+0000 7f4149693640 10 librados: watch_flush enter 2024-03-29T11:53:44.429 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.421+0000 7f4149693640 10 librados: watch_flush exit 2024-03-29T11:53:44.429 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.421+0000 7f4149693640 20 client.4432.objecter shutdown clearing up homeless session... 2024-03-29T11:53:44.430 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.421+0000 7f4149693640 10 client.4432.objecter successfully canceled tick 2024-03-29T11:53:44.430 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.422+0000 7f4149693640 1 -- 172.21.15.98:0/808924995 >> v1:172.21.15.98:6816/4065186974 conn(0x7f412406de10 legacy=0x7f41240702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:44.430 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.422+0000 7f4149693640 10 monclient: shutdown 2024-03-29T11:53:44.430 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.422+0000 7f4149693640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:53:44.430 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.422+0000 7f4149693640 1 -- 172.21.15.98:0/808924995 >> v1:172.21.15.121:6789/0 conn(0x7f414413bad0 legacy=0x7f41440713b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:44.430 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.422+0000 7f4149693640 1 -- 172.21.15.98:0/808924995 shutdown_connections 2024-03-29T11:53:44.430 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.422+0000 7f4149693640 1 -- 172.21.15.98:0/808924995 >> 172.21.15.98:0/808924995 conn(0x7f41440b3c50 msgr2=0x7f41440b4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:53:44.431 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.423+0000 7f4149693640 1 -- 172.21.15.98:0/808924995 shutdown_connections 2024-03-29T11:53:44.431 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.423+0000 7f4149693640 1 -- 172.21.15.98:0/808924995 wait complete. 2024-03-29T11:53:44.433 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:44.423+0000 7f4149693640 1 librados: shutdown 2024-03-29T11:53:44.450 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:53:51.257 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:53:51.417 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.409+0000 7f1837806640 10 monclient: get_monmap_and_config 2024-03-29T11:53:51.417 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.409+0000 7f1837806640 10 monclient: build_initial_monmap 2024-03-29T11:53:51.417 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.409+0000 7f1837806640 10 monclient: monmap: 2024-03-29T11:53:51.417 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:53:51.417 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:51.417 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:53:51.410571+0000 2024-03-29T11:53:51.417 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:53:51.410571+0000 2024-03-29T11:53:51.418 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:53:51.418 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:51.418 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:53:51.418 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:53:51.418 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:53:51.418 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:51.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.410+0000 7f1837806640 1 Processor -- start 2024-03-29T11:53:51.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.410+0000 7f1837806640 1 -- start start 2024-03-29T11:53:51.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.410+0000 7f1837806640 10 monclient: init 2024-03-29T11:53:51.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.411+0000 7f1837806640 10 monclient: _reopen_session rank -1 2024-03-29T11:53:51.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.411+0000 7f1837806640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:53:51.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.411+0000 7f1837806640 10 monclient(hunting): picked mon.noname-b con 0x7f18301422b0 addr v1:172.21.15.98:6790/0 2024-03-29T11:53:51.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.411+0000 7f1837806640 10 monclient(hunting): picked mon.noname-c con 0x7f1830142690 addr v1:172.21.15.121:6789/0 2024-03-29T11:53:51.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.411+0000 7f1837806640 10 monclient(hunting): picked mon.noname-a con 0x7f18300b17d0 addr v1:172.21.15.98:6789/0 2024-03-29T11:53:51.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.411+0000 7f1837806640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1830142c80 con 0x7f18300b17d0 2024-03-29T11:53:51.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.411+0000 7f1837806640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18300aae40 con 0x7f1830142690 2024-03-29T11:53:51.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.411+0000 7f1837806640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f183013fbd0 con 0x7f18301422b0 2024-03-29T11:53:51.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.411+0000 7f1837806640 10 monclient(hunting): _renew_subs 2024-03-29T11:53:51.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.411+0000 7f1837806640 10 monclient(hunting): authenticate will time out at 1138.496795s 2024-03-29T11:53:51.420 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.412+0000 7f1835da3640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f18300b17d0 0x7f18300b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56308/0 (socket says 172.21.15.98:56308) 2024-03-29T11:53:51.420 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.412+0000 7f1835da3640 1 -- 172.21.15.98:0/3328224113 learned_addr learned my addr 172.21.15.98:0/3328224113 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:53:51.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.413+0000 7f1827fff640 1 -- 172.21.15.98:0/3328224113 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3371170669 0 0) 0x7f1830142c80 con 0x7f18300b17d0 2024-03-29T11:53:51.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.413+0000 7f1827fff640 10 monclient(hunting): _negotiate 2024-03-29T11:53:51.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.413+0000 7f1827fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:51.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.413+0000 7f1827fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:51.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.413+0000 7f1827fff640 10 monclient(hunting): my global_id is 4410 2024-03-29T11:53:51.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.413+0000 7f1827fff640 1 -- 172.21.15.98:0/3328224113 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f181c003520 con 0x7f18300b17d0 2024-03-29T11:53:51.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.413+0000 7f1827fff640 1 -- 172.21.15.98:0/3328224113 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3383908192 0 0) 0x7f183013fbd0 con 0x7f18301422b0 2024-03-29T11:53:51.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.413+0000 7f1827fff640 10 monclient(hunting): _negotiate 2024-03-29T11:53:51.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.413+0000 7f1827fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:51.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.413+0000 7f1827fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:51.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.413+0000 7f1827fff640 10 monclient(hunting): my global_id is 4439 2024-03-29T11:53:51.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.413+0000 7f1827fff640 1 -- 172.21.15.98:0/3328224113 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1830142c80 con 0x7f18301422b0 2024-03-29T11:53:51.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.414+0000 7f1827fff640 1 -- 172.21.15.98:0/3328224113 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1015491842 0 0) 0x7f1830142c80 con 0x7f18301422b0 2024-03-29T11:53:51.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.414+0000 7f1827fff640 1 -- 172.21.15.98:0/3328224113 >> v1:172.21.15.121:6789/0 conn(0x7f1830142690 legacy=0x7f18300b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:53:51.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.414+0000 7f1827fff640 1 -- 172.21.15.98:0/3328224113 >> v1:172.21.15.98:6789/0 conn(0x7f18300b17d0 legacy=0x7f18300b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:51.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.414+0000 7f1827fff640 10 monclient: _finish_hunting 0 2024-03-29T11:53:51.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.414+0000 7f1827fff640 1 monclient: found mon.noname-b 2024-03-29T11:53:51.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.414+0000 7f1827fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:53:51.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.414+0000 7f1827fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:53:51.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.414+0000 7f1827fff640 1 -- 172.21.15.98:0/3328224113 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1830151940 con 0x7f18301422b0 2024-03-29T11:53:51.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.414+0000 7f1827fff640 10 monclient: _finish_auth 0 2024-03-29T11:53:51.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.414+0000 7f1827fff640 10 monclient: _check_auth_tickets 2024-03-29T11:53:51.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.414+0000 7f1827fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:53:51.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.414+0000 7f1827fff640 1 -- 172.21.15.98:0/3328224113 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f18280018a0 con 0x7f18301422b0 2024-03-29T11:53:51.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.414+0000 7f1827fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:51.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.414+0000 7f1827fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:53:51.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.414+0000 7f1827fff640 10 monclient: dump: 2024-03-29T11:53:51.425 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:53:51.425 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:51.425 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:51.425 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:51.425 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:51.425 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:51.425 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:53:51.425 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:53:51.425 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:53:51.425 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:51.425 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.414+0000 7f1837806640 5 monclient: authenticate success, global_id 4439 2024-03-29T11:53:51.425 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.414+0000 7f1837806640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:53:51.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.416+0000 7f1827fff640 1 -- 172.21.15.98:0/3328224113 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1828001a00 con 0x7f18301422b0 2024-03-29T11:53:51.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.416+0000 7f1827fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:53:51.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.416+0000 7f1827fff640 1 -- 172.21.15.98:0/3328224113 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f1828001dd0 con 0x7f18301422b0 2024-03-29T11:53:51.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.416+0000 7f1827fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:51.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.416+0000 7f1827fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:53:51.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.416+0000 7f1827fff640 10 monclient: dump: 2024-03-29T11:53:51.427 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:53:51.427 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:51.427 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:51.427 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:51.427 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:51.427 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:51.427 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:53:51.428 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:53:51.428 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:53:51.428 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:51.428 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.416+0000 7f1837806640 10 monclient: get_monmap_and_config success 2024-03-29T11:53:51.428 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.416+0000 7f1837806640 4 set_mon_vals no callback set 2024-03-29T11:53:51.428 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.417+0000 7f1837806640 10 monclient: shutdown 2024-03-29T11:53:51.428 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.417+0000 7f1837806640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:53:51.428 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.417+0000 7f1837806640 1 -- 172.21.15.98:0/3328224113 >> v1:172.21.15.98:6790/0 conn(0x7f18301422b0 legacy=0x7f183013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:51.428 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.418+0000 7f1837806640 1 -- 172.21.15.98:0/3328224113 shutdown_connections 2024-03-29T11:53:51.428 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.418+0000 7f1837806640 1 -- 172.21.15.98:0/3328224113 >> 172.21.15.98:0/3328224113 conn(0x7f1830066a00 msgr2=0x7f1830066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:53:51.428 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.418+0000 7f1837806640 1 -- 172.21.15.98:0/3328224113 shutdown_connections 2024-03-29T11:53:51.428 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.419+0000 7f1837806640 1 -- 172.21.15.98:0/3328224113 wait complete. 2024-03-29T11:53:51.429 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.421+0000 7f1837806640 10 monclient: build_initial_monmap 2024-03-29T11:53:51.429 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.421+0000 7f1837806640 10 monclient: monmap: 2024-03-29T11:53:51.429 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:53:51.429 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:53:51.429 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:53:51.429 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:53:51.429 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:53:51.429 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:51.429 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:53:51.429 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:53:51.429 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:53:51.429 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:51.430 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.421+0000 7f1837806640 1 librados: starting msgr at 2024-03-29T11:53:51.430 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.421+0000 7f1837806640 1 librados: starting objecter 2024-03-29T11:53:51.430 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.422+0000 7f1837806640 1 Processor -- start 2024-03-29T11:53:51.430 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.422+0000 7f1837806640 1 -- start start 2024-03-29T11:53:51.430 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.422+0000 7f1837806640 1 librados: setting wanted keys 2024-03-29T11:53:51.430 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.422+0000 7f1837806640 1 librados: calling monclient init 2024-03-29T11:53:51.430 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.422+0000 7f1837806640 10 monclient: init 2024-03-29T11:53:51.433 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.423+0000 7f1837806640 10 monclient: _reopen_session rank -1 2024-03-29T11:53:51.433 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.423+0000 7f1837806640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:53:51.433 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.423+0000 7f1837806640 10 monclient(hunting): picked mon.noname-b con 0x7f18300b17d0 addr v1:172.21.15.121:6789/0 2024-03-29T11:53:51.433 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.423+0000 7f1837806640 10 monclient(hunting): picked mon.noname-c con 0x7f18301422b0 addr v1:172.21.15.98:6790/0 2024-03-29T11:53:51.433 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.423+0000 7f1837806640 10 monclient(hunting): picked mon.noname-a con 0x7f1830142690 addr v1:172.21.15.98:6789/0 2024-03-29T11:53:51.433 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.423+0000 7f1837806640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18300b6680 con 0x7f1830142690 2024-03-29T11:53:51.433 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.423+0000 7f1837806640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f183006f280 con 0x7f18300b17d0 2024-03-29T11:53:51.433 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.423+0000 7f1837806640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f183006f560 con 0x7f18301422b0 2024-03-29T11:53:51.433 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.423+0000 7f1837806640 10 monclient(hunting): _renew_subs 2024-03-29T11:53:51.433 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.423+0000 7f1837806640 10 monclient(hunting): authenticate will time out at 1138.508547s 2024-03-29T11:53:51.433 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.424+0000 7f1835da3640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f1830142690 0x7f18301ddf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56562/0 (socket says 172.21.15.98:56562) 2024-03-29T11:53:51.433 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.424+0000 7f1835da3640 1 -- 172.21.15.98:0/4238762742 learned_addr learned my addr 172.21.15.98:0/4238762742 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:53:51.433 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.425+0000 7f1825ffb640 10 client.?.objecter ms_handle_connect 0x7f1830142690 2024-03-29T11:53:51.433 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.425+0000 7f1825ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:53:51.434 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.425+0000 7f1825ffb640 1 -- 172.21.15.98:0/4238762742 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3369690852 0 0) 0x7f18300b6680 con 0x7f1830142690 2024-03-29T11:53:51.434 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.425+0000 7f1825ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:53:51.434 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.425+0000 7f1825ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:51.434 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.425+0000 7f1825ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:51.434 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.425+0000 7f1825ffb640 10 monclient(hunting): my global_id is 4413 2024-03-29T11:53:51.434 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.425+0000 7f1825ffb640 1 -- 172.21.15.98:0/4238762742 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1814004eb0 con 0x7f1830142690 2024-03-29T11:53:51.434 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.425+0000 7f1825ffb640 10 client.?.objecter ms_handle_connect 0x7f18301422b0 2024-03-29T11:53:51.434 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.425+0000 7f1825ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:53:51.436 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.425+0000 7f1825ffb640 1 -- 172.21.15.98:0/4238762742 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2036505491 0 0) 0x7f1814004eb0 con 0x7f1830142690 2024-03-29T11:53:51.436 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.426+0000 7f1825ffb640 1 -- 172.21.15.98:0/4238762742 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18300b6680 con 0x7f1830142690 2024-03-29T11:53:51.436 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.426+0000 7f1825ffb640 1 -- 172.21.15.98:0/4238762742 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f1820001910 con 0x7f1830142690 2024-03-29T11:53:51.436 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.426+0000 7f1825ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:51.436 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.426+0000 7f1825ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:53:51.436 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.426+0000 7f1825ffb640 10 monclient(hunting): dump: 2024-03-29T11:53:51.436 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:53:51.436 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:51.436 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:51.436 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:51.436 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:51.436 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:51.436 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:53:51.436 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:53:51.436 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:53:51.437 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:51.437 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.426+0000 7f1825ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:53:51.437 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.426+0000 7f1825ffb640 1 -- 172.21.15.98:0/4238762742 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3281161706 0 0) 0x7f18300b6680 con 0x7f1830142690 2024-03-29T11:53:51.437 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.426+0000 7f1825ffb640 1 -- 172.21.15.98:0/4238762742 >> v1:172.21.15.98:6790/0 conn(0x7f18301422b0 legacy=0x7f18301da800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:51.437 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.426+0000 7f1825ffb640 1 -- 172.21.15.98:0/4238762742 >> v1:172.21.15.121:6789/0 conn(0x7f18300b17d0 legacy=0x7f18300b5560 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:53:51.437 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.426+0000 7f1825ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:53:51.437 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.426+0000 7f1825ffb640 1 monclient: found mon.a 2024-03-29T11:53:51.437 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.426+0000 7f1825ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:53:51.437 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.426+0000 7f1825ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:53:51.437 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.426+0000 7f1825ffb640 1 -- 172.21.15.98:0/4238762742 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f183006e320 con 0x7f1830142690 2024-03-29T11:53:51.437 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.426+0000 7f1825ffb640 10 monclient: _finish_auth 0 2024-03-29T11:53:51.437 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.426+0000 7f1825ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:53:51.437 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.426+0000 7f1825ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:53:51.437 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.426+0000 7f1837806640 5 monclient: authenticate success, global_id 4413 2024-03-29T11:53:51.437 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.426+0000 7f1837806640 10 monclient: _renew_subs 2024-03-29T11:53:51.437 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.426+0000 7f1837806640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:53:51.438 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.426+0000 7f1837806640 1 -- 172.21.15.98:0/4238762742 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f183012f5a0 con 0x7f1830142690 2024-03-29T11:53:51.438 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.427+0000 7f1837806640 10 client.4413.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:53:51.438 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.427+0000 7f1837806640 10 monclient: _renew_subs 2024-03-29T11:53:51.438 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.427+0000 7f1837806640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:53:51.438 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.427+0000 7f1837806640 1 -- 172.21.15.98:0/4238762742 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f183006e070 con 0x7f1830142690 2024-03-29T11:53:51.438 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.427+0000 7f1837806640 1 librados: init done 2024-03-29T11:53:51.438 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.428+0000 7f1825ffb640 1 -- 172.21.15.98:0/4238762742 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18200047e0 con 0x7f1830142690 2024-03-29T11:53:51.438 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.428+0000 7f1825ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:53:51.438 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.428+0000 7f1825ffb640 1 -- 172.21.15.98:0/4238762742 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f1820004bf0 con 0x7f1830142690 2024-03-29T11:53:51.438 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.428+0000 7f1825ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:51.438 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.428+0000 7f1825ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:53:51.439 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.428+0000 7f1825ffb640 10 monclient: dump: 2024-03-29T11:53:51.439 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:53:51.439 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:51.439 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:51.439 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:51.439 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:51.439 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:51.439 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:53:51.439 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:53:51.439 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:53:51.439 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:51.439 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.429+0000 7f1837806640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:53:51.439 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.429+0000 7f1837806640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:53:51.439 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.429+0000 7f1837806640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:53:51.439 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.429+0000 7f1837806640 1 -- 172.21.15.98:0/4238762742 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1830075f10 con 0x7f1830142690 2024-03-29T11:53:51.439 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.430+0000 7f1825ffb640 1 -- 172.21.15.98:0/4238762742 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f1820004d50 con 0x7f1830142690 2024-03-29T11:53:51.440 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.431+0000 7f18277fe640 4 set_mon_vals no callback set 2024-03-29T11:53:51.440 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.431+0000 7f18355a2640 10 monclient: get_auth_request con 0x7f181406dd00 auth_method 0 2024-03-29T11:53:51.440 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.432+0000 7f1825ffb640 10 client.4413.objecter ms_handle_connect 0x7f181406dd00 2024-03-29T11:53:51.441 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.432+0000 7f1825ffb640 1 -- 172.21.15.98:0/4238762742 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f1820084a90 con 0x7f1830142690 2024-03-29T11:53:51.441 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.432+0000 7f1825ffb640 10 client.4413.objecter ms_dispatch 0x7f1830000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:53:51.441 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.432+0000 7f1825ffb640 3 client.4413.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:53:51.441 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.432+0000 7f1825ffb640 3 client.4413.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:53:51.441 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.432+0000 7f1825ffb640 20 client.4413.objecter dump_active .. 0 homeless 2024-03-29T11:53:51.448 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.441+0000 7f1825ffb640 1 -- 172.21.15.98:0/4238762742 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f18200503d0 con 0x7f1830142690 2024-03-29T11:53:51.448 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.441+0000 7f1825ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:53:51.448 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.441+0000 7f1825ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:53:51.810 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.802+0000 7f1837806640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:53:51.810 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.802+0000 7f1837806640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:53:51.810 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.802+0000 7f1837806640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:53:51.810 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.802+0000 7f1837806640 1 -- 172.21.15.98:0/4238762742 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18300b2a30 con 0x7f1830142690 2024-03-29T11:53:51.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.804+0000 7f1825ffb640 1 -- 172.21.15.98:0/4238762742 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f1820052300 con 0x7f1830142690 2024-03-29T11:53:51.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.804+0000 7f1825ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:53:51.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.804+0000 7f1825ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:53:51.812 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:53:51.812 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:53:51.815 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.808+0000 7f1837806640 10 librados: watch_flush enter 2024-03-29T11:53:51.817 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.810+0000 7f1837806640 10 librados: watch_flush exit 2024-03-29T11:53:51.817 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.810+0000 7f1837806640 20 client.4413.objecter shutdown clearing up homeless session... 2024-03-29T11:53:51.817 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.810+0000 7f1837806640 10 client.4413.objecter successfully canceled tick 2024-03-29T11:53:51.818 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.811+0000 7f1837806640 1 -- 172.21.15.98:0/4238762742 >> v1:172.21.15.98:6816/4065186974 conn(0x7f181406dd00 legacy=0x7f18140701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:51.818 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.811+0000 7f1837806640 10 monclient: shutdown 2024-03-29T11:53:51.818 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.811+0000 7f1837806640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:53:51.819 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.811+0000 7f1837806640 1 -- 172.21.15.98:0/4238762742 >> v1:172.21.15.98:6789/0 conn(0x7f1830142690 legacy=0x7f18301ddf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:51.819 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.811+0000 7f1837806640 1 -- 172.21.15.98:0/4238762742 shutdown_connections 2024-03-29T11:53:51.819 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.811+0000 7f1837806640 1 -- 172.21.15.98:0/4238762742 >> 172.21.15.98:0/4238762742 conn(0x7f1830066a00 msgr2=0x7f183013c1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:53:51.819 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.811+0000 7f1837806640 1 -- 172.21.15.98:0/4238762742 shutdown_connections 2024-03-29T11:53:51.819 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.811+0000 7f1837806640 1 -- 172.21.15.98:0/4238762742 wait complete. 2024-03-29T11:53:51.819 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:51.812+0000 7f1837806640 1 librados: shutdown 2024-03-29T11:53:51.835 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:53:58.643 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:53:58.790 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.782+0000 7f54d973b640 10 monclient: get_monmap_and_config 2024-03-29T11:53:58.791 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.782+0000 7f54d973b640 10 monclient: build_initial_monmap 2024-03-29T11:53:58.791 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.783+0000 7f54d973b640 10 monclient: monmap: 2024-03-29T11:53:58.791 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:53:58.791 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:58.791 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:53:58.783908+0000 2024-03-29T11:53:58.791 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:53:58.783908+0000 2024-03-29T11:53:58.791 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:53:58.791 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:58.791 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:53:58.791 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:53:58.791 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:53:58.791 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:58.792 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.783+0000 7f54d973b640 1 Processor -- start 2024-03-29T11:53:58.793 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.784+0000 7f54d973b640 1 -- start start 2024-03-29T11:53:58.793 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.784+0000 7f54d973b640 10 monclient: init 2024-03-29T11:53:58.793 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.785+0000 7f54d973b640 10 monclient: _reopen_session rank -1 2024-03-29T11:53:58.793 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.785+0000 7f54d973b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:53:58.793 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.785+0000 7f54d973b640 10 monclient(hunting): picked mon.noname-c con 0x7f54d41457f0 addr v1:172.21.15.121:6789/0 2024-03-29T11:53:58.793 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.785+0000 7f54d973b640 10 monclient(hunting): picked mon.noname-b con 0x7f54d413f650 addr v1:172.21.15.98:6790/0 2024-03-29T11:53:58.793 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.785+0000 7f54d973b640 10 monclient(hunting): picked mon.noname-a con 0x7f54d40a9d70 addr v1:172.21.15.98:6789/0 2024-03-29T11:53:58.793 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.785+0000 7f54d973b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54d40aaef0 con 0x7f54d40a9d70 2024-03-29T11:53:58.793 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.785+0000 7f54d973b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54d4145bd0 con 0x7f54d41457f0 2024-03-29T11:53:58.793 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.785+0000 7f54d973b640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54d4145eb0 con 0x7f54d413f650 2024-03-29T11:53:58.793 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.785+0000 7f54d973b640 10 monclient(hunting): _renew_subs 2024-03-29T11:53:58.793 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.785+0000 7f54d973b640 10 monclient(hunting): authenticate will time out at 1145.870375s 2024-03-29T11:53:58.793 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.785+0000 7f54ca7fc640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f54d413f650 0x7f54d40b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:54914/0 (socket says 172.21.15.98:54914) 2024-03-29T11:53:58.794 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.785+0000 7f54ca7fc640 1 -- 172.21.15.98:0/2647496903 learned_addr learned my addr 172.21.15.98:0/2647496903 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:53:58.795 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.786+0000 7f54d27fc640 1 -- 172.21.15.98:0/2647496903 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3200798651 0 0) 0x7f54d4145eb0 con 0x7f54d413f650 2024-03-29T11:53:58.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.786+0000 7f54d27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:53:58.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.786+0000 7f54d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:58.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.786+0000 7f54d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:58.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.786+0000 7f54d27fc640 10 monclient(hunting): my global_id is 4442 2024-03-29T11:53:58.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.786+0000 7f54d27fc640 1 -- 172.21.15.98:0/2647496903 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54b4003520 con 0x7f54d413f650 2024-03-29T11:53:58.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.786+0000 7f54d27fc640 1 -- 172.21.15.98:0/2647496903 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3944132190 0 0) 0x7f54b4003520 con 0x7f54d413f650 2024-03-29T11:53:58.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.786+0000 7f54d27fc640 1 -- 172.21.15.98:0/2647496903 >> v1:172.21.15.121:6789/0 conn(0x7f54d41457f0 legacy=0x7f54d413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:53:58.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.786+0000 7f54d27fc640 1 -- 172.21.15.98:0/2647496903 >> v1:172.21.15.98:6789/0 conn(0x7f54d40a9d70 legacy=0x7f54d414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:58.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.786+0000 7f54d27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:53:58.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.786+0000 7f54d27fc640 1 monclient: found mon.noname-b 2024-03-29T11:53:58.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.786+0000 7f54d27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:53:58.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.786+0000 7f54d27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:53:58.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.786+0000 7f54d27fc640 1 -- 172.21.15.98:0/2647496903 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d4146340 con 0x7f54d413f650 2024-03-29T11:53:58.797 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.786+0000 7f54d27fc640 10 monclient: _finish_auth 0 2024-03-29T11:53:58.797 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.786+0000 7f54d27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:53:58.797 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.786+0000 7f54d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:53:58.797 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.786+0000 7f54d27fc640 1 -- 172.21.15.98:0/2647496903 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f54bc001a10 con 0x7f54d413f650 2024-03-29T11:53:58.797 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.786+0000 7f54d973b640 5 monclient: authenticate success, global_id 4442 2024-03-29T11:53:58.797 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.786+0000 7f54d973b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:53:58.797 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.786+0000 7f54d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:58.797 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.786+0000 7f54d27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:53:58.797 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.786+0000 7f54d27fc640 10 monclient: dump: 2024-03-29T11:53:58.797 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:53:58.797 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:58.797 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:58.797 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:58.797 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:58.797 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:58.798 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:53:58.798 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:53:58.798 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:53:58.798 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:58.798 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.786+0000 7f54d973b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:53:58.798 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.787+0000 7f54d27fc640 1 -- 172.21.15.98:0/2647496903 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54bc001be0 con 0x7f54d413f650 2024-03-29T11:53:58.798 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.787+0000 7f54d27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:53:58.798 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.787+0000 7f54d27fc640 1 -- 172.21.15.98:0/2647496903 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f54bc002030 con 0x7f54d413f650 2024-03-29T11:53:58.798 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.787+0000 7f54d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:58.798 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.787+0000 7f54d27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:53:58.798 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.787+0000 7f54d27fc640 10 monclient: dump: 2024-03-29T11:53:58.798 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:53:58.798 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:58.798 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:58.799 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:58.799 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:58.799 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:58.799 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:53:58.799 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:53:58.799 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:53:58.799 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:58.799 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.787+0000 7f54d973b640 10 monclient: get_monmap_and_config success 2024-03-29T11:53:58.799 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.787+0000 7f54d973b640 4 set_mon_vals no callback set 2024-03-29T11:53:58.799 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.787+0000 7f54d973b640 10 monclient: shutdown 2024-03-29T11:53:58.799 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.787+0000 7f54d973b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:53:58.799 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.787+0000 7f54d973b640 1 -- 172.21.15.98:0/2647496903 >> v1:172.21.15.98:6790/0 conn(0x7f54d413f650 legacy=0x7f54d40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:58.799 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.788+0000 7f54d973b640 1 -- 172.21.15.98:0/2647496903 shutdown_connections 2024-03-29T11:53:58.799 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.788+0000 7f54d973b640 1 -- 172.21.15.98:0/2647496903 >> 172.21.15.98:0/2647496903 conn(0x7f54d4066a00 msgr2=0x7f54d4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:53:58.799 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.788+0000 7f54d973b640 1 -- 172.21.15.98:0/2647496903 shutdown_connections 2024-03-29T11:53:58.799 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.788+0000 7f54d973b640 1 -- 172.21.15.98:0/2647496903 wait complete. 2024-03-29T11:53:58.805 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.789+0000 7f54d973b640 10 monclient: build_initial_monmap 2024-03-29T11:53:58.805 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.789+0000 7f54d973b640 10 monclient: monmap: 2024-03-29T11:53:58.805 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:53:58.805 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:53:58.805 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:53:58.805 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:53:58.806 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:53:58.806 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:58.806 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:53:58.806 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:53:58.806 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:53:58.806 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:58.806 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.789+0000 7f54d973b640 1 librados: starting msgr at 2024-03-29T11:53:58.806 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.789+0000 7f54d973b640 1 librados: starting objecter 2024-03-29T11:53:58.806 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.789+0000 7f54d973b640 1 Processor -- start 2024-03-29T11:53:58.806 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.790+0000 7f54d973b640 1 -- start start 2024-03-29T11:53:58.806 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.790+0000 7f54d973b640 1 librados: setting wanted keys 2024-03-29T11:53:58.806 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.790+0000 7f54d973b640 1 librados: calling monclient init 2024-03-29T11:53:58.806 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.790+0000 7f54d973b640 10 monclient: init 2024-03-29T11:53:58.806 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.791+0000 7f54d973b640 10 monclient: _reopen_session rank -1 2024-03-29T11:53:58.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.791+0000 7f54d973b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:53:58.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.791+0000 7f54d973b640 10 monclient(hunting): picked mon.noname-b con 0x7f54d40a9d70 addr v1:172.21.15.121:6789/0 2024-03-29T11:53:58.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.791+0000 7f54d973b640 10 monclient(hunting): picked mon.noname-c con 0x7f54d413f650 addr v1:172.21.15.98:6790/0 2024-03-29T11:53:58.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.791+0000 7f54ca7fc640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f54d413f650 0x7f54d4142080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:55978/0 (socket says 172.21.15.98:55978) 2024-03-29T11:53:58.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.791+0000 7f54ca7fc640 1 -- 172.21.15.98:0/2388926046 learned_addr learned my addr 172.21.15.98:0/2388926046 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:53:58.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.791+0000 7f54cbfff640 10 client.?.objecter ms_handle_connect 0x7f54d413f650 2024-03-29T11:53:58.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.791+0000 7f54cbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:53:58.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.791+0000 7f54d973b640 10 monclient(hunting): picked mon.noname-a con 0x7f54d41457f0 addr v1:172.21.15.98:6789/0 2024-03-29T11:53:58.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.792+0000 7f54d973b640 1 -- 172.21.15.98:0/2388926046 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54d4141780 con 0x7f54d41457f0 2024-03-29T11:53:58.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.792+0000 7f54cbfff640 10 client.?.objecter ms_handle_connect 0x7f54d40a9d70 2024-03-29T11:53:58.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.792+0000 7f54cbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:53:58.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.792+0000 7f54cbfff640 10 client.?.objecter ms_handle_connect 0x7f54d41457f0 2024-03-29T11:53:58.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.792+0000 7f54cbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:53:58.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.792+0000 7f54d973b640 1 -- 172.21.15.98:0/2388926046 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54d4141910 con 0x7f54d40a9d70 2024-03-29T11:53:58.808 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.792+0000 7f54d973b640 1 -- 172.21.15.98:0/2388926046 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54d4139ff0 con 0x7f54d413f650 2024-03-29T11:53:58.808 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.792+0000 7f54d973b640 10 monclient(hunting): _renew_subs 2024-03-29T11:53:58.808 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.792+0000 7f54cbfff640 1 -- 172.21.15.98:0/2388926046 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 837234024 0 0) 0x7f54d4141780 con 0x7f54d41457f0 2024-03-29T11:53:58.808 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.792+0000 7f54d973b640 10 monclient(hunting): authenticate will time out at 1145.877799s 2024-03-29T11:53:58.808 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.792+0000 7f54cbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:53:58.808 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.792+0000 7f54cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:58.808 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.792+0000 7f54cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:58.808 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.792+0000 7f54cbfff640 10 monclient(hunting): my global_id is 4419 2024-03-29T11:53:58.808 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.792+0000 7f54cbfff640 1 -- 172.21.15.98:0/2388926046 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54b8004eb0 con 0x7f54d41457f0 2024-03-29T11:53:58.808 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.792+0000 7f54cbfff640 1 -- 172.21.15.98:0/2388926046 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2659549274 0 0) 0x7f54d4141910 con 0x7f54d40a9d70 2024-03-29T11:53:58.808 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.792+0000 7f54cbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:53:58.808 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.792+0000 7f54cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:58.808 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.792+0000 7f54cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:58.808 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.792+0000 7f54cbfff640 10 monclient(hunting): my global_id is 4435 2024-03-29T11:53:58.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.792+0000 7f54cbfff640 1 -- 172.21.15.98:0/2388926046 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54d4141780 con 0x7f54d40a9d70 2024-03-29T11:53:58.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.793+0000 7f54cbfff640 1 -- 172.21.15.98:0/2388926046 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3294926468 0 0) 0x7f54d4139ff0 con 0x7f54d413f650 2024-03-29T11:53:58.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.793+0000 7f54cbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:53:58.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.793+0000 7f54cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:53:58.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.793+0000 7f54cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:53:58.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.793+0000 7f54cbfff640 10 monclient(hunting): my global_id is 4445 2024-03-29T11:53:58.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.793+0000 7f54cbfff640 1 -- 172.21.15.98:0/2388926046 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54d4141910 con 0x7f54d413f650 2024-03-29T11:53:58.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.793+0000 7f54cbfff640 1 -- 172.21.15.98:0/2388926046 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1068723434 0 0) 0x7f54b8004eb0 con 0x7f54d41457f0 2024-03-29T11:53:58.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.793+0000 7f54cbfff640 1 -- 172.21.15.98:0/2388926046 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54d4139ff0 con 0x7f54d41457f0 2024-03-29T11:53:58.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.793+0000 7f54cbfff640 1 -- 172.21.15.98:0/2388926046 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f54cc0018b0 con 0x7f54d41457f0 2024-03-29T11:53:58.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.793+0000 7f54cbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:58.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.793+0000 7f54cbfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:53:58.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.793+0000 7f54cbfff640 10 monclient(hunting): dump: 2024-03-29T11:53:58.810 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:53:58.810 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:58.810 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:58.810 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:58.810 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:58.810 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:58.810 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:53:58.810 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:53:58.810 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:53:58.810 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:58.810 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.793+0000 7f54cbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:53:58.810 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.793+0000 7f54cbfff640 1 -- 172.21.15.98:0/2388926046 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3030530486 0 0) 0x7f54d4141780 con 0x7f54d40a9d70 2024-03-29T11:53:58.810 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.793+0000 7f54cbfff640 1 -- 172.21.15.98:0/2388926046 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54b8006bb0 con 0x7f54d40a9d70 2024-03-29T11:53:58.810 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.793+0000 7f54cbfff640 1 -- 172.21.15.98:0/2388926046 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f54c4001770 con 0x7f54d40a9d70 2024-03-29T11:53:58.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.793+0000 7f54cbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:58.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.793+0000 7f54cbfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:53:58.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.793+0000 7f54cbfff640 10 monclient(hunting): dump: 2024-03-29T11:53:58.811 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:53:58.811 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:58.811 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:58.811 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:58.811 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:58.811 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:58.811 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:53:58.811 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:53:58.811 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:53:58.811 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:58.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.793+0000 7f54cbfff640 1 -- 172.21.15.98:0/2388926046 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 902177305 0 0) 0x7f54d4141910 con 0x7f54d413f650 2024-03-29T11:53:58.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.793+0000 7f54cbfff640 1 -- 172.21.15.98:0/2388926046 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54d4141780 con 0x7f54d413f650 2024-03-29T11:53:58.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.793+0000 7f54cbfff640 1 -- 172.21.15.98:0/2388926046 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f54bc001ef0 con 0x7f54d413f650 2024-03-29T11:53:58.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.793+0000 7f54cbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:58.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.793+0000 7f54cbfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:53:58.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.793+0000 7f54cbfff640 10 monclient(hunting): dump: 2024-03-29T11:53:58.812 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:53:58.812 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:58.812 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:58.812 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:58.812 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:58.812 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:58.812 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:53:58.812 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:53:58.812 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:53:58.812 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:58.813 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.794+0000 7f54cbfff640 1 -- 172.21.15.98:0/2388926046 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3254777493 0 0) 0x7f54d4141780 con 0x7f54d413f650 2024-03-29T11:53:58.813 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.794+0000 7f54cbfff640 1 -- 172.21.15.98:0/2388926046 >> v1:172.21.15.121:6789/0 conn(0x7f54d40a9d70 legacy=0x7f54d413ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:58.813 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.794+0000 7f54cbfff640 1 -- 172.21.15.98:0/2388926046 >> v1:172.21.15.98:6789/0 conn(0x7f54d41457f0 legacy=0x7f54d4142790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:58.813 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.794+0000 7f54cbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:53:58.813 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.794+0000 7f54cbfff640 1 monclient: found mon.c 2024-03-29T11:53:58.813 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.794+0000 7f54cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:53:58.813 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.794+0000 7f54cbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:53:58.813 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.794+0000 7f54cbfff640 1 -- 172.21.15.98:0/2388926046 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d413a180 con 0x7f54d413f650 2024-03-29T11:53:58.813 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.794+0000 7f54cbfff640 10 monclient: _finish_auth 0 2024-03-29T11:53:58.813 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.794+0000 7f54cbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:53:58.813 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.794+0000 7f54cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:53:58.813 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.794+0000 7f54d973b640 5 monclient: authenticate success, global_id 4445 2024-03-29T11:53:58.813 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.794+0000 7f54d973b640 10 monclient: _renew_subs 2024-03-29T11:53:58.813 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.794+0000 7f54d973b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:53:58.813 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.794+0000 7f54d973b640 1 -- 172.21.15.98:0/2388926046 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54d413a370 con 0x7f54d413f650 2024-03-29T11:53:58.814 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.794+0000 7f54d973b640 10 client.4445.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:53:58.814 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.794+0000 7f54d973b640 10 monclient: _renew_subs 2024-03-29T11:53:58.814 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.794+0000 7f54d973b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:53:58.814 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.794+0000 7f54d973b640 1 -- 172.21.15.98:0/2388926046 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f54d41e7360 con 0x7f54d413f650 2024-03-29T11:53:58.814 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.794+0000 7f54d973b640 1 librados: init done 2024-03-29T11:53:58.814 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.794+0000 7f54cbfff640 1 -- 172.21.15.98:0/2388926046 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54bc004f70 con 0x7f54d413f650 2024-03-29T11:53:58.814 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.794+0000 7f54cbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:53:58.814 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.794+0000 7f54cbfff640 1 -- 172.21.15.98:0/2388926046 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f54bc005280 con 0x7f54d413f650 2024-03-29T11:53:58.814 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.794+0000 7f54cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:53:58.814 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.794+0000 7f54cbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:53:58.814 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.794+0000 7f54cbfff640 10 monclient: dump: 2024-03-29T11:53:58.814 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:53:58.814 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:53:58.814 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:58.815 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:53:58.815 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:53:58.815 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:53:58.815 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:53:58.815 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:53:58.815 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:53:58.815 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:53:58.815 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.794+0000 7f54d1ffb640 4 set_mon_vals no callback set 2024-03-29T11:53:58.815 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.795+0000 7f54d973b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:53:58.815 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.795+0000 7f54d973b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:53:58.815 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.795+0000 7f54d973b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:53:58.815 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.795+0000 7f54d973b640 1 -- 172.21.15.98:0/2388926046 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54ac005080 con 0x7f54d413f650 2024-03-29T11:53:58.815 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.796+0000 7f54cbfff640 1 -- 172.21.15.98:0/2388926046 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f54bc01b250 con 0x7f54d413f650 2024-03-29T11:53:58.815 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.797+0000 7f54cbfff640 1 -- 172.21.15.98:0/2388926046 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f54bc084e30 con 0x7f54d413f650 2024-03-29T11:53:58.815 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.797+0000 7f54cbfff640 10 client.4445.objecter ms_dispatch 0x7f54d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:53:58.816 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.797+0000 7f54cbfff640 3 client.4445.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:53:58.816 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.797+0000 7f54cbfff640 3 client.4445.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:53:58.816 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.797+0000 7f54cbfff640 20 client.4445.objecter dump_active .. 0 homeless 2024-03-29T11:53:58.816 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.797+0000 7f54d2ffd640 10 monclient: get_auth_request con 0x7f54b806e4c0 auth_method 0 2024-03-29T11:53:58.816 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.798+0000 7f54cbfff640 10 client.4445.objecter ms_handle_connect 0x7f54b806e4c0 2024-03-29T11:53:58.816 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.802+0000 7f54cbfff640 1 -- 172.21.15.98:0/2388926046 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f54bc050710 con 0x7f54d413f650 2024-03-29T11:53:58.816 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.802+0000 7f54cbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:53:58.817 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:58.802+0000 7f54cbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:53:59.171 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:59.164+0000 7f54d973b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:53:59.172 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:59.164+0000 7f54d973b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:53:59.172 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:59.164+0000 7f54d973b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:53:59.172 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:59.164+0000 7f54d973b640 1 -- 172.21.15.98:0/2388926046 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54ac005330 con 0x7f54d413f650 2024-03-29T11:53:59.173 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:59.165+0000 7f54cbfff640 1 -- 172.21.15.98:0/2388926046 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f54bc052640 con 0x7f54d413f650 2024-03-29T11:53:59.173 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:59.165+0000 7f54cbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:53:59.173 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:59.165+0000 7f54cbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:53:59.174 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:53:59.174 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:53:59.178 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:59.170+0000 7f54d973b640 10 librados: watch_flush enter 2024-03-29T11:53:59.178 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:59.170+0000 7f54d973b640 10 librados: watch_flush exit 2024-03-29T11:53:59.179 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:59.170+0000 7f54d973b640 20 client.4445.objecter shutdown clearing up homeless session... 2024-03-29T11:53:59.179 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:59.170+0000 7f54d973b640 10 client.4445.objecter successfully canceled tick 2024-03-29T11:53:59.179 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:59.170+0000 7f54d973b640 1 -- 172.21.15.98:0/2388926046 >> v1:172.21.15.98:6816/4065186974 conn(0x7f54b806e4c0 legacy=0x7f54b8070980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:59.179 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:59.170+0000 7f54d973b640 10 monclient: shutdown 2024-03-29T11:53:59.179 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:59.170+0000 7f54d973b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:53:59.179 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:59.170+0000 7f54d973b640 1 -- 172.21.15.98:0/2388926046 >> v1:172.21.15.98:6790/0 conn(0x7f54d413f650 legacy=0x7f54d4142080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:53:59.179 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:59.170+0000 7f54d973b640 1 -- 172.21.15.98:0/2388926046 shutdown_connections 2024-03-29T11:53:59.179 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:59.170+0000 7f54d973b640 1 -- 172.21.15.98:0/2388926046 >> 172.21.15.98:0/2388926046 conn(0x7f54d4066a00 msgr2=0x7f54d41d2420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:53:59.179 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:59.171+0000 7f54d973b640 1 -- 172.21.15.98:0/2388926046 shutdown_connections 2024-03-29T11:53:59.179 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:59.171+0000 7f54d973b640 1 -- 172.21.15.98:0/2388926046 wait complete. 2024-03-29T11:53:59.179 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:53:59.171+0000 7f54d973b640 1 librados: shutdown 2024-03-29T11:53:59.194 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:54:06.001 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:54:06.135 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.127+0000 7f2f62838640 10 monclient: get_monmap_and_config 2024-03-29T11:54:06.135 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.127+0000 7f2f62838640 10 monclient: build_initial_monmap 2024-03-29T11:54:06.136 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.127+0000 7f2f62838640 10 monclient: monmap: 2024-03-29T11:54:06.136 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:54:06.136 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:06.136 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:54:06.128829+0000 2024-03-29T11:54:06.136 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:54:06.128829+0000 2024-03-29T11:54:06.136 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:54:06.136 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:06.136 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:54:06.136 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:54:06.136 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:54:06.136 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:06.137 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.128+0000 7f2f62838640 1 Processor -- start 2024-03-29T11:54:06.137 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.128+0000 7f2f62838640 1 -- start start 2024-03-29T11:54:06.137 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.128+0000 7f2f62838640 10 monclient: init 2024-03-29T11:54:06.138 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.130+0000 7f2f62838640 10 monclient: _reopen_session rank -1 2024-03-29T11:54:06.138 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.130+0000 7f2f62838640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:54:06.138 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.130+0000 7f2f62838640 10 monclient(hunting): picked mon.noname-a con 0x7f2f5c06c250 addr v1:172.21.15.98:6789/0 2024-03-29T11:54:06.139 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.130+0000 7f2f62838640 10 monclient(hunting): picked mon.noname-b con 0x7f2f5c0ae650 addr v1:172.21.15.98:6790/0 2024-03-29T11:54:06.139 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.130+0000 7f2f62838640 10 monclient(hunting): picked mon.noname-c con 0x7f2f5c13f8c0 addr v1:172.21.15.121:6789/0 2024-03-29T11:54:06.139 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.130+0000 7f2f62838640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f5c0adf50 con 0x7f2f5c06c250 2024-03-29T11:54:06.139 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.130+0000 7f2f62838640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f5c0ae230 con 0x7f2f5c13f8c0 2024-03-29T11:54:06.139 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.130+0000 7f2f62838640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f5c13fd50 con 0x7f2f5c0ae650 2024-03-29T11:54:06.139 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.130+0000 7f2f62838640 10 monclient(hunting): _renew_subs 2024-03-29T11:54:06.139 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.130+0000 7f2f62838640 10 monclient(hunting): authenticate will time out at 1153.215993s 2024-03-29T11:54:06.139 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.131+0000 7f2f60dd5640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f2f5c13f8c0 0x7f2f5c14db30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:51044/0 (socket says 172.21.15.98:51044) 2024-03-29T11:54:06.139 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.131+0000 7f2f60dd5640 1 -- 172.21.15.98:0/3410177761 learned_addr learned my addr 172.21.15.98:0/3410177761 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:54:06.140 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.132+0000 7f2f5affd640 1 -- 172.21.15.98:0/3410177761 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1726660562 0 0) 0x7f2f5c0ae230 con 0x7f2f5c13f8c0 2024-03-29T11:54:06.141 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.132+0000 7f2f5affd640 10 monclient(hunting): _negotiate 2024-03-29T11:54:06.141 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.132+0000 7f2f5affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:54:06.141 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.132+0000 7f2f5affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:54:06.141 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.132+0000 7f2f5affd640 10 monclient(hunting): my global_id is 4438 2024-03-29T11:54:06.141 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.132+0000 7f2f5affd640 1 -- 172.21.15.98:0/3410177761 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f44003520 con 0x7f2f5c13f8c0 2024-03-29T11:54:06.141 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.133+0000 7f2f5affd640 1 -- 172.21.15.98:0/3410177761 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3615101951 0 0) 0x7f2f5c13fd50 con 0x7f2f5c0ae650 2024-03-29T11:54:06.141 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.133+0000 7f2f5affd640 10 monclient(hunting): _negotiate 2024-03-29T11:54:06.141 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.133+0000 7f2f5affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:54:06.141 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.133+0000 7f2f5affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:54:06.141 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.133+0000 7f2f5affd640 10 monclient(hunting): my global_id is 4448 2024-03-29T11:54:06.141 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.133+0000 7f2f5affd640 1 -- 172.21.15.98:0/3410177761 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f5c0ae230 con 0x7f2f5c0ae650 2024-03-29T11:54:06.141 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.133+0000 7f2f5affd640 1 -- 172.21.15.98:0/3410177761 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 464037431 0 0) 0x7f2f44003520 con 0x7f2f5c13f8c0 2024-03-29T11:54:06.141 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.133+0000 7f2f5affd640 1 -- 172.21.15.98:0/3410177761 >> v1:172.21.15.98:6790/0 conn(0x7f2f5c0ae650 legacy=0x7f2f5c13f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:06.141 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.133+0000 7f2f5affd640 1 -- 172.21.15.98:0/3410177761 >> v1:172.21.15.98:6789/0 conn(0x7f2f5c06c250 legacy=0x7f2f5c1427b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:54:06.143 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.134+0000 7f2f5affd640 10 monclient: _finish_hunting 0 2024-03-29T11:54:06.143 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.134+0000 7f2f5affd640 1 monclient: found mon.noname-c 2024-03-29T11:54:06.143 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.134+0000 7f2f5affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:54:06.143 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.134+0000 7f2f5affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:54:06.143 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.134+0000 7f2f5affd640 1 -- 172.21.15.98:0/3410177761 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f5c06db20 con 0x7f2f5c13f8c0 2024-03-29T11:54:06.143 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.134+0000 7f2f5affd640 10 monclient: _finish_auth 0 2024-03-29T11:54:06.143 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.134+0000 7f2f5affd640 10 monclient: _check_auth_tickets 2024-03-29T11:54:06.143 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.134+0000 7f2f5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:54:06.143 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.134+0000 7f2f5affd640 1 -- 172.21.15.98:0/3410177761 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f2f54001a10 con 0x7f2f5c13f8c0 2024-03-29T11:54:06.144 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.134+0000 7f2f5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:54:06.144 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.134+0000 7f2f5affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:54:06.144 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.134+0000 7f2f5affd640 10 monclient: dump: 2024-03-29T11:54:06.144 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:54:06.144 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:06.144 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:06.144 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:06.144 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:54:06.144 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:06.144 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:54:06.144 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:54:06.144 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:54:06.144 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:06.144 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.134+0000 7f2f62838640 5 monclient: authenticate success, global_id 4438 2024-03-29T11:54:06.144 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.134+0000 7f2f62838640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:54:06.145 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.134+0000 7f2f5affd640 1 -- 172.21.15.98:0/3410177761 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f54001b70 con 0x7f2f5c13f8c0 2024-03-29T11:54:06.145 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.134+0000 7f2f5affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:54:06.145 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.134+0000 7f2f5affd640 1 -- 172.21.15.98:0/3410177761 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f2f54001f80 con 0x7f2f5c13f8c0 2024-03-29T11:54:06.145 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.134+0000 7f2f5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:54:06.145 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.134+0000 7f2f5affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:54:06.145 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.134+0000 7f2f5affd640 10 monclient: dump: 2024-03-29T11:54:06.145 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:54:06.145 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:06.145 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:06.145 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:06.145 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:54:06.145 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:06.145 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:54:06.145 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:54:06.145 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:54:06.145 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:06.146 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.134+0000 7f2f62838640 10 monclient: get_monmap_and_config success 2024-03-29T11:54:06.146 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.134+0000 7f2f62838640 4 set_mon_vals no callback set 2024-03-29T11:54:06.146 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.135+0000 7f2f62838640 10 monclient: shutdown 2024-03-29T11:54:06.146 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.135+0000 7f2f62838640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:54:06.146 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.135+0000 7f2f62838640 1 -- 172.21.15.98:0/3410177761 >> v1:172.21.15.121:6789/0 conn(0x7f2f5c13f8c0 legacy=0x7f2f5c14db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:06.153 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.136+0000 7f2f62838640 1 -- 172.21.15.98:0/3410177761 shutdown_connections 2024-03-29T11:54:06.153 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.136+0000 7f2f62838640 1 -- 172.21.15.98:0/3410177761 >> 172.21.15.98:0/3410177761 conn(0x7f2f5c067610 msgr2=0x7f2f5c0b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:54:06.154 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.136+0000 7f2f62838640 1 -- 172.21.15.98:0/3410177761 shutdown_connections 2024-03-29T11:54:06.154 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.136+0000 7f2f62838640 1 -- 172.21.15.98:0/3410177761 wait complete. 2024-03-29T11:54:06.154 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.137+0000 7f2f62838640 10 monclient: build_initial_monmap 2024-03-29T11:54:06.154 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.137+0000 7f2f62838640 10 monclient: monmap: 2024-03-29T11:54:06.154 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:54:06.154 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:54:06.154 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:54:06.154 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:54:06.154 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:54:06.154 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:06.154 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:54:06.154 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:54:06.155 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:54:06.155 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:06.155 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.137+0000 7f2f62838640 1 librados: starting msgr at 2024-03-29T11:54:06.155 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.137+0000 7f2f62838640 1 librados: starting objecter 2024-03-29T11:54:06.155 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.137+0000 7f2f62838640 1 Processor -- start 2024-03-29T11:54:06.155 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.138+0000 7f2f62838640 1 -- start start 2024-03-29T11:54:06.155 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.138+0000 7f2f62838640 1 librados: setting wanted keys 2024-03-29T11:54:06.155 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.138+0000 7f2f62838640 1 librados: calling monclient init 2024-03-29T11:54:06.155 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.138+0000 7f2f62838640 10 monclient: init 2024-03-29T11:54:06.155 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.139+0000 7f2f62838640 10 monclient: _reopen_session rank -1 2024-03-29T11:54:06.155 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.139+0000 7f2f62838640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:54:06.155 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.139+0000 7f2f62838640 10 monclient(hunting): picked mon.noname-a con 0x7f2f5c06c250 addr v1:172.21.15.98:6789/0 2024-03-29T11:54:06.156 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.139+0000 7f2f62838640 10 monclient(hunting): picked mon.noname-c con 0x7f2f5c0ae650 addr v1:172.21.15.98:6790/0 2024-03-29T11:54:06.156 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.139+0000 7f2f5b7fe640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f2f5c0ae650 0x7f2f5c135f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:52346/0 (socket says 172.21.15.98:52346) 2024-03-29T11:54:06.156 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.139+0000 7f2f62838640 10 monclient(hunting): picked mon.noname-b con 0x7f2f5c13f8c0 addr v1:172.21.15.121:6789/0 2024-03-29T11:54:06.156 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.139+0000 7f2f62838640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f5c070000 con 0x7f2f5c06c250 2024-03-29T11:54:06.156 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.139+0000 7f2f5b7fe640 1 -- 172.21.15.98:0/1713476002 learned_addr learned my addr 172.21.15.98:0/1713476002 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:54:06.156 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.139+0000 7f2f62838640 1 -- 172.21.15.98:0/1713476002 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f5c136d30 con 0x7f2f5c13f8c0 2024-03-29T11:54:06.156 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.139+0000 7f2f62838640 1 -- 172.21.15.98:0/1713476002 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f5c072380 con 0x7f2f5c0ae650 2024-03-29T11:54:06.156 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.139+0000 7f2f62838640 10 monclient(hunting): _renew_subs 2024-03-29T11:54:06.156 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.139+0000 7f2f58ff9640 10 client.?.objecter ms_handle_connect 0x7f2f5c0ae650 2024-03-29T11:54:06.156 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.139+0000 7f2f58ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:54:06.156 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.139+0000 7f2f62838640 10 monclient(hunting): authenticate will time out at 1153.224967s 2024-03-29T11:54:06.156 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.140+0000 7f2f58ff9640 10 client.?.objecter ms_handle_connect 0x7f2f5c13f8c0 2024-03-29T11:54:06.157 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.140+0000 7f2f58ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:54:06.157 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.140+0000 7f2f58ff9640 1 -- 172.21.15.98:0/1713476002 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 415643655 0 0) 0x7f2f5c072380 con 0x7f2f5c0ae650 2024-03-29T11:54:06.157 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.140+0000 7f2f58ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:54:06.157 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.140+0000 7f2f58ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:54:06.157 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.140+0000 7f2f58ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:54:06.157 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.140+0000 7f2f58ff9640 10 monclient(hunting): my global_id is 4451 2024-03-29T11:54:06.157 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.140+0000 7f2f58ff9640 1 -- 172.21.15.98:0/1713476002 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f48004eb0 con 0x7f2f5c0ae650 2024-03-29T11:54:06.157 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.140+0000 7f2f58ff9640 1 -- 172.21.15.98:0/1713476002 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2842538092 0 0) 0x7f2f5c136d30 con 0x7f2f5c13f8c0 2024-03-29T11:54:06.157 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.140+0000 7f2f58ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:54:06.157 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.140+0000 7f2f58ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:54:06.157 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.140+0000 7f2f58ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:54:06.157 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.140+0000 7f2f58ff9640 10 monclient(hunting): my global_id is 4441 2024-03-29T11:54:06.157 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.140+0000 7f2f58ff9640 1 -- 172.21.15.98:0/1713476002 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f5c072380 con 0x7f2f5c13f8c0 2024-03-29T11:54:06.157 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.140+0000 7f2f58ff9640 1 -- 172.21.15.98:0/1713476002 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2586783121 0 0) 0x7f2f48004eb0 con 0x7f2f5c0ae650 2024-03-29T11:54:06.157 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.141+0000 7f2f58ff9640 1 -- 172.21.15.98:0/1713476002 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f5c136d30 con 0x7f2f5c0ae650 2024-03-29T11:54:06.158 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.141+0000 7f2f58ff9640 1 -- 172.21.15.98:0/1713476002 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f2f50001a90 con 0x7f2f5c0ae650 2024-03-29T11:54:06.158 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.141+0000 7f2f58ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:54:06.158 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.141+0000 7f2f58ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:54:06.158 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.141+0000 7f2f58ff9640 10 monclient(hunting): dump: 2024-03-29T11:54:06.158 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:54:06.158 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:06.158 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:06.158 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:06.158 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:54:06.158 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:06.159 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:54:06.159 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:54:06.159 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:54:06.159 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:06.159 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.141+0000 7f2f58ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:54:06.159 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.141+0000 7f2f58ff9640 1 -- 172.21.15.98:0/1713476002 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 987274756 0 0) 0x7f2f5c072380 con 0x7f2f5c13f8c0 2024-03-29T11:54:06.159 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.141+0000 7f2f58ff9640 1 -- 172.21.15.98:0/1713476002 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f48004eb0 con 0x7f2f5c13f8c0 2024-03-29T11:54:06.159 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.141+0000 7f2f58ff9640 1 -- 172.21.15.98:0/1713476002 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f2f54001e80 con 0x7f2f5c13f8c0 2024-03-29T11:54:06.159 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.141+0000 7f2f58ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:54:06.159 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.141+0000 7f2f58ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:54:06.159 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.141+0000 7f2f58ff9640 10 monclient(hunting): dump: 2024-03-29T11:54:06.159 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:54:06.160 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:06.160 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:06.160 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:06.160 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:54:06.160 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:06.160 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:54:06.160 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:54:06.160 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:54:06.160 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:06.160 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.141+0000 7f2f58ff9640 1 -- 172.21.15.98:0/1713476002 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2742876964 0 0) 0x7f2f5c136d30 con 0x7f2f5c0ae650 2024-03-29T11:54:06.160 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.141+0000 7f2f58ff9640 1 -- 172.21.15.98:0/1713476002 >> v1:172.21.15.121:6789/0 conn(0x7f2f5c13f8c0 legacy=0x7f2f5c136620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:06.160 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.141+0000 7f2f58ff9640 1 -- 172.21.15.98:0/1713476002 >> v1:172.21.15.98:6789/0 conn(0x7f2f5c06c250 legacy=0x7f2f5c139250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:54:06.160 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.142+0000 7f2f58ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:54:06.160 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.142+0000 7f2f58ff9640 1 monclient: found mon.c 2024-03-29T11:54:06.160 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.142+0000 7f2f58ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:54:06.161 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.142+0000 7f2f58ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:54:06.161 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.142+0000 7f2f58ff9640 1 -- 172.21.15.98:0/1713476002 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f5c070490 con 0x7f2f5c0ae650 2024-03-29T11:54:06.161 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.142+0000 7f2f58ff9640 10 monclient: _finish_auth 0 2024-03-29T11:54:06.161 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.142+0000 7f2f58ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:54:06.161 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.142+0000 7f2f58ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:54:06.161 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.142+0000 7f2f62838640 5 monclient: authenticate success, global_id 4451 2024-03-29T11:54:06.161 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.142+0000 7f2f62838640 10 monclient: _renew_subs 2024-03-29T11:54:06.161 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.142+0000 7f2f62838640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:54:06.161 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.142+0000 7f2f62838640 1 -- 172.21.15.98:0/1713476002 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f5c137500 con 0x7f2f5c0ae650 2024-03-29T11:54:06.161 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.142+0000 7f2f62838640 10 client.4451.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:54:06.161 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.142+0000 7f2f62838640 10 monclient: _renew_subs 2024-03-29T11:54:06.161 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.142+0000 7f2f62838640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:54:06.161 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.142+0000 7f2f62838640 1 -- 172.21.15.98:0/1713476002 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f5c071390 con 0x7f2f5c0ae650 2024-03-29T11:54:06.161 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.142+0000 7f2f62838640 1 librados: init done 2024-03-29T11:54:06.162 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.143+0000 7f2f58ff9640 1 -- 172.21.15.98:0/1713476002 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f50001d50 con 0x7f2f5c0ae650 2024-03-29T11:54:06.162 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.143+0000 7f2f58ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:54:06.162 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.143+0000 7f2f58ff9640 1 -- 172.21.15.98:0/1713476002 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f2f50003a60 con 0x7f2f5c0ae650 2024-03-29T11:54:06.162 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.143+0000 7f2f58ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:54:06.162 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.143+0000 7f2f58ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:54:06.162 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.143+0000 7f2f58ff9640 10 monclient: dump: 2024-03-29T11:54:06.162 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:54:06.162 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:06.162 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:06.162 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:06.162 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:54:06.162 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:06.162 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:54:06.163 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:54:06.163 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:54:06.163 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:06.163 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.143+0000 7f2f5a7fc640 4 set_mon_vals no callback set 2024-03-29T11:54:06.163 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.144+0000 7f2f62838640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:54:06.163 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.144+0000 7f2f62838640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:54:06.163 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.144+0000 7f2f62838640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:54:06.163 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.144+0000 7f2f62838640 1 -- 172.21.15.98:0/1713476002 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f3c005080 con 0x7f2f5c0ae650 2024-03-29T11:54:06.164 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.146+0000 7f2f58ff9640 1 -- 172.21.15.98:0/1713476002 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f2f50003bc0 con 0x7f2f5c0ae650 2024-03-29T11:54:06.164 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.147+0000 7f2f58ff9640 1 -- 172.21.15.98:0/1713476002 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f2f50084c00 con 0x7f2f5c0ae650 2024-03-29T11:54:06.164 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.147+0000 7f2f58ff9640 10 client.4451.objecter ms_dispatch 0x7f2f5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:54:06.165 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.147+0000 7f2f58ff9640 3 client.4451.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:54:06.165 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.147+0000 7f2f58ff9640 3 client.4451.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:54:06.165 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.147+0000 7f2f58ff9640 20 client.4451.objecter dump_active .. 0 homeless 2024-03-29T11:54:06.165 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.147+0000 7f2f5bfff640 10 monclient: get_auth_request con 0x7f2f4806de80 auth_method 0 2024-03-29T11:54:06.165 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.147+0000 7f2f58ff9640 10 client.4451.objecter ms_handle_connect 0x7f2f4806de80 2024-03-29T11:54:06.165 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.155+0000 7f2f58ff9640 1 -- 172.21.15.98:0/1713476002 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2f50050310 con 0x7f2f5c0ae650 2024-03-29T11:54:06.165 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.155+0000 7f2f58ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:54:06.165 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.155+0000 7f2f58ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:54:06.469 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.461+0000 7f2f62838640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:54:06.469 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.461+0000 7f2f62838640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:54:06.469 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.461+0000 7f2f62838640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:54:06.469 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.461+0000 7f2f62838640 1 -- 172.21.15.98:0/1713476002 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f3c005330 con 0x7f2f5c0ae650 2024-03-29T11:54:06.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.462+0000 7f2f58ff9640 1 -- 172.21.15.98:0/1713476002 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f2f50052410 con 0x7f2f5c0ae650 2024-03-29T11:54:06.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.462+0000 7f2f58ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:54:06.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.462+0000 7f2f58ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:54:06.470 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:54:06.470 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:54:06.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.465+0000 7f2f62838640 10 librados: watch_flush enter 2024-03-29T11:54:06.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.465+0000 7f2f62838640 10 librados: watch_flush exit 2024-03-29T11:54:06.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.465+0000 7f2f62838640 20 client.4451.objecter shutdown clearing up homeless session... 2024-03-29T11:54:06.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.465+0000 7f2f62838640 10 client.4451.objecter successfully canceled tick 2024-03-29T11:54:06.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.466+0000 7f2f62838640 1 -- 172.21.15.98:0/1713476002 >> v1:172.21.15.98:6816/4065186974 conn(0x7f2f4806de80 legacy=0x7f2f48070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:06.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.466+0000 7f2f62838640 10 monclient: shutdown 2024-03-29T11:54:06.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.466+0000 7f2f62838640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:54:06.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.466+0000 7f2f62838640 1 -- 172.21.15.98:0/1713476002 >> v1:172.21.15.98:6790/0 conn(0x7f2f5c0ae650 legacy=0x7f2f5c135f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:06.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.466+0000 7f2f62838640 1 -- 172.21.15.98:0/1713476002 shutdown_connections 2024-03-29T11:54:06.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.466+0000 7f2f62838640 1 -- 172.21.15.98:0/1713476002 >> 172.21.15.98:0/1713476002 conn(0x7f2f5c067610 msgr2=0x7f2f5c1427b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:54:06.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.466+0000 7f2f62838640 1 -- 172.21.15.98:0/1713476002 shutdown_connections 2024-03-29T11:54:06.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.466+0000 7f2f62838640 1 -- 172.21.15.98:0/1713476002 wait complete. 2024-03-29T11:54:06.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:06.466+0000 7f2f62838640 1 librados: shutdown 2024-03-29T11:54:06.488 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:54:13.295 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:54:13.444 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.435+0000 7f6e052ba640 10 monclient: get_monmap_and_config 2024-03-29T11:54:13.444 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.435+0000 7f6e052ba640 10 monclient: build_initial_monmap 2024-03-29T11:54:13.444 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.435+0000 7f6e052ba640 10 monclient: monmap: 2024-03-29T11:54:13.444 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:54:13.444 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:13.444 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:54:13.437079+0000 2024-03-29T11:54:13.444 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:54:13.437079+0000 2024-03-29T11:54:13.444 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:54:13.444 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:13.444 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:54:13.444 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:54:13.445 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:54:13.445 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:13.446 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.438+0000 7f6e052ba640 1 Processor -- start 2024-03-29T11:54:13.446 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.438+0000 7f6e052ba640 1 -- start start 2024-03-29T11:54:13.447 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.438+0000 7f6e052ba640 10 monclient: init 2024-03-29T11:54:13.447 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.439+0000 7f6e052ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:54:13.447 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.439+0000 7f6e052ba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:54:13.449 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.439+0000 7f6e052ba640 10 monclient(hunting): picked mon.noname-b con 0x7f6df8038c60 addr v1:172.21.15.98:6790/0 2024-03-29T11:54:13.449 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.439+0000 7f6e052ba640 10 monclient(hunting): picked mon.noname-c con 0x7f6df80324b0 addr v1:172.21.15.121:6789/0 2024-03-29T11:54:13.449 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.440+0000 7f6dfffff640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f6df8038c60 0x7f6df80c4ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:48734/0 (socket says 172.21.15.98:48734) 2024-03-29T11:54:13.449 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.440+0000 7f6e052ba640 10 monclient(hunting): picked mon.noname-a con 0x7f6df80d11e0 addr v1:172.21.15.98:6789/0 2024-03-29T11:54:13.449 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.440+0000 7f6e052ba640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6df80caf80 con 0x7f6df80d11e0 2024-03-29T11:54:13.449 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.440+0000 7f6e052ba640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6df8039850 con 0x7f6df80324b0 2024-03-29T11:54:13.449 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.440+0000 7f6e052ba640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6df80c78d0 con 0x7f6df8038c60 2024-03-29T11:54:13.449 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.440+0000 7f6e052ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:54:13.449 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.440+0000 7f6e052ba640 10 monclient(hunting): authenticate will time out at 1160.526402s 2024-03-29T11:54:13.449 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.440+0000 7f6dfffff640 1 -- 172.21.15.98:0/3857616442 learned_addr learned my addr 172.21.15.98:0/3857616442 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:54:13.451 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.442+0000 7f6dfeffd640 1 -- 172.21.15.98:0/3857616442 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3446798996 0 0) 0x7f6df80c78d0 con 0x7f6df8038c60 2024-03-29T11:54:13.451 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.442+0000 7f6dfeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:54:13.452 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.442+0000 7f6dfeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:54:13.452 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.442+0000 7f6dfeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:54:13.452 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.442+0000 7f6dfeffd640 10 monclient(hunting): my global_id is 4454 2024-03-29T11:54:13.452 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.442+0000 7f6dfeffd640 1 -- 172.21.15.98:0/3857616442 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6df0003520 con 0x7f6df8038c60 2024-03-29T11:54:13.452 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.443+0000 7f6dfeffd640 1 -- 172.21.15.98:0/3857616442 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1053258915 0 0) 0x7f6df80caf80 con 0x7f6df80d11e0 2024-03-29T11:54:13.452 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.443+0000 7f6dfeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:54:13.452 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.443+0000 7f6dfeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:54:13.452 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.443+0000 7f6dfeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:54:13.452 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.443+0000 7f6dfeffd640 10 monclient(hunting): my global_id is 4422 2024-03-29T11:54:13.452 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.443+0000 7f6dfeffd640 1 -- 172.21.15.98:0/3857616442 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6df80c78d0 con 0x7f6df80d11e0 2024-03-29T11:54:13.454 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.444+0000 7f6dfeffd640 1 -- 172.21.15.98:0/3857616442 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4007685844 0 0) 0x7f6df0003520 con 0x7f6df8038c60 2024-03-29T11:54:13.454 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.444+0000 7f6dfeffd640 1 -- 172.21.15.98:0/3857616442 >> v1:172.21.15.121:6789/0 conn(0x7f6df80324b0 legacy=0x7f6df80cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:13.454 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.444+0000 7f6dfeffd640 1 -- 172.21.15.98:0/3857616442 >> v1:172.21.15.98:6789/0 conn(0x7f6df80d11e0 legacy=0x7f6df80d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:13.454 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.444+0000 7f6dfeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:54:13.454 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.444+0000 7f6dfeffd640 1 monclient: found mon.noname-b 2024-03-29T11:54:13.454 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.444+0000 7f6dfeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:54:13.454 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.444+0000 7f6dfeffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:54:13.454 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.444+0000 7f6dfeffd640 1 -- 172.21.15.98:0/3857616442 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df80c7a60 con 0x7f6df8038c60 2024-03-29T11:54:13.455 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.444+0000 7f6dfeffd640 10 monclient: _finish_auth 0 2024-03-29T11:54:13.455 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.444+0000 7f6dfeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:54:13.455 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.444+0000 7f6dfeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:54:13.455 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.444+0000 7f6dfeffd640 1 -- 172.21.15.98:0/3857616442 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f6dec001690 con 0x7f6df8038c60 2024-03-29T11:54:13.455 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.444+0000 7f6dfeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:54:13.455 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.444+0000 7f6dfeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:54:13.455 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.444+0000 7f6dfeffd640 10 monclient: dump: 2024-03-29T11:54:13.455 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:54:13.455 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:13.455 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:13.455 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:13.455 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:54:13.455 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:13.455 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:54:13.456 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:54:13.456 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:54:13.456 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:13.456 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.444+0000 7f6e052ba640 5 monclient: authenticate success, global_id 4454 2024-03-29T11:54:13.456 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.444+0000 7f6e052ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:54:13.456 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.444+0000 7f6dfeffd640 1 -- 172.21.15.98:0/3857616442 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6dec001860 con 0x7f6df8038c60 2024-03-29T11:54:13.456 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.444+0000 7f6dfeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:54:13.456 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.444+0000 7f6dfeffd640 1 -- 172.21.15.98:0/3857616442 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f6dec001cb0 con 0x7f6df8038c60 2024-03-29T11:54:13.456 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.444+0000 7f6dfeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:54:13.456 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.444+0000 7f6dfeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:54:13.456 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.444+0000 7f6dfeffd640 10 monclient: dump: 2024-03-29T11:54:13.456 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:54:13.456 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:13.456 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:13.457 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:13.457 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:54:13.457 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:13.457 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:54:13.457 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:54:13.457 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:54:13.457 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:13.457 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.444+0000 7f6e052ba640 10 monclient: get_monmap_and_config success 2024-03-29T11:54:13.457 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.444+0000 7f6e052ba640 4 set_mon_vals no callback set 2024-03-29T11:54:13.457 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.445+0000 7f6e052ba640 10 monclient: shutdown 2024-03-29T11:54:13.457 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.445+0000 7f6e052ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:54:13.457 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.445+0000 7f6e052ba640 1 -- 172.21.15.98:0/3857616442 >> v1:172.21.15.98:6790/0 conn(0x7f6df8038c60 legacy=0x7f6df80c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:13.457 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.445+0000 7f6e052ba640 1 -- 172.21.15.98:0/3857616442 shutdown_connections 2024-03-29T11:54:13.457 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.445+0000 7f6e052ba640 1 -- 172.21.15.98:0/3857616442 >> 172.21.15.98:0/3857616442 conn(0x7f6df8005c20 msgr2=0x7f6df8006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:54:13.459 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.446+0000 7f6e052ba640 1 -- 172.21.15.98:0/3857616442 shutdown_connections 2024-03-29T11:54:13.459 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.446+0000 7f6e052ba640 1 -- 172.21.15.98:0/3857616442 wait complete. 2024-03-29T11:54:13.459 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.447+0000 7f6e052ba640 10 monclient: build_initial_monmap 2024-03-29T11:54:13.459 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.447+0000 7f6e052ba640 10 monclient: monmap: 2024-03-29T11:54:13.459 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:54:13.459 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:54:13.459 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:54:13.460 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:54:13.460 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:54:13.460 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:13.460 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:54:13.460 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:54:13.460 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:54:13.460 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:13.460 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.448+0000 7f6e052ba640 1 librados: starting msgr at 2024-03-29T11:54:13.460 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.448+0000 7f6e052ba640 1 librados: starting objecter 2024-03-29T11:54:13.460 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.448+0000 7f6e052ba640 1 Processor -- start 2024-03-29T11:54:13.460 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.448+0000 7f6e052ba640 1 -- start start 2024-03-29T11:54:13.460 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.448+0000 7f6e052ba640 1 librados: setting wanted keys 2024-03-29T11:54:13.460 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.448+0000 7f6e052ba640 1 librados: calling monclient init 2024-03-29T11:54:13.460 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.449+0000 7f6e052ba640 10 monclient: init 2024-03-29T11:54:13.460 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.450+0000 7f6e052ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:54:13.461 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.450+0000 7f6e052ba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:54:13.461 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.450+0000 7f6e052ba640 10 monclient(hunting): picked mon.noname-b con 0x7f6df80324b0 addr v1:172.21.15.121:6789/0 2024-03-29T11:54:13.462 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.454+0000 7f6e052ba640 10 monclient(hunting): picked mon.noname-a con 0x7f6df8038c60 addr v1:172.21.15.98:6789/0 2024-03-29T11:54:13.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.454+0000 7f6dff7fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f6df8038c60 0x7f6df80b7dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:49303/0 (socket says 172.21.15.98:49303) 2024-03-29T11:54:13.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.454+0000 7f6e052ba640 10 monclient(hunting): picked mon.noname-c con 0x7f6df80d11e0 addr v1:172.21.15.98:6790/0 2024-03-29T11:54:13.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.454+0000 7f6dff7fe640 1 -- 172.21.15.98:0/3258645663 learned_addr learned my addr 172.21.15.98:0/3258645663 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:54:13.469 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.454+0000 7f6e052ba640 1 -- 172.21.15.98:0/3258645663 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6df80b8bf0 con 0x7f6df8038c60 2024-03-29T11:54:13.469 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.454+0000 7f6e052ba640 1 -- 172.21.15.98:0/3258645663 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6df8036110 con 0x7f6df80324b0 2024-03-29T11:54:13.469 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.454+0000 7f6dfcff9640 10 client.?.objecter ms_handle_connect 0x7f6df8038c60 2024-03-29T11:54:13.469 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.454+0000 7f6dfcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:54:13.469 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.455+0000 7f6dfcff9640 1 -- 172.21.15.98:0/3258645663 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 827439437 0 0) 0x7f6df80b8bf0 con 0x7f6df8038c60 2024-03-29T11:54:13.469 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.455+0000 7f6e052ba640 1 -- 172.21.15.98:0/3258645663 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6df80363f0 con 0x7f6df80d11e0 2024-03-29T11:54:13.469 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.455+0000 7f6e052ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:54:13.469 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.455+0000 7f6e052ba640 10 monclient(hunting): authenticate will time out at 1160.541613s 2024-03-29T11:54:13.469 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.455+0000 7f6dfcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:54:13.469 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.455+0000 7f6dfcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:54:13.469 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.455+0000 7f6dfcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:54:13.469 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.455+0000 7f6dfcff9640 10 monclient(hunting): my global_id is 4425 2024-03-29T11:54:13.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.455+0000 7f6dfcff9640 1 -- 172.21.15.98:0/3258645663 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6de4004eb0 con 0x7f6df8038c60 2024-03-29T11:54:13.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.455+0000 7f6dfcff9640 10 client.?.objecter ms_handle_connect 0x7f6df80d11e0 2024-03-29T11:54:13.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.455+0000 7f6dfcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:54:13.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.456+0000 7f6dfcff9640 1 -- 172.21.15.98:0/3258645663 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1771359568 0 0) 0x7f6df80363f0 con 0x7f6df80d11e0 2024-03-29T11:54:13.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.456+0000 7f6dfcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:54:13.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.456+0000 7f6dfcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:54:13.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.456+0000 7f6dfcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:54:13.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.456+0000 7f6dfcff9640 10 monclient(hunting): my global_id is 4457 2024-03-29T11:54:13.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.456+0000 7f6dfcff9640 1 -- 172.21.15.98:0/3258645663 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6df80b8bf0 con 0x7f6df80d11e0 2024-03-29T11:54:13.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.456+0000 7f6dfcff9640 10 client.?.objecter ms_handle_connect 0x7f6df80324b0 2024-03-29T11:54:13.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.456+0000 7f6dfcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:54:13.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.456+0000 7f6dfcff9640 1 -- 172.21.15.98:0/3258645663 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1805719835 0 0) 0x7f6de4004eb0 con 0x7f6df8038c60 2024-03-29T11:54:13.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.456+0000 7f6dfcff9640 1 -- 172.21.15.98:0/3258645663 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6df80363f0 con 0x7f6df8038c60 2024-03-29T11:54:13.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.457+0000 7f6dfcff9640 1 -- 172.21.15.98:0/3258645663 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f6df4001a10 con 0x7f6df8038c60 2024-03-29T11:54:13.471 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.457+0000 7f6dfcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:54:13.471 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.457+0000 7f6dfcff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:54:13.471 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.457+0000 7f6dfcff9640 10 monclient(hunting): dump: 2024-03-29T11:54:13.471 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:54:13.471 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:13.471 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:13.471 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:13.471 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:54:13.471 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:13.471 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:54:13.471 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:54:13.471 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:54:13.471 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:13.471 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.457+0000 7f6dfcff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:54:13.471 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.457+0000 7f6dfcff9640 1 -- 172.21.15.98:0/3258645663 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1187898003 0 0) 0x7f6df80b8bf0 con 0x7f6df80d11e0 2024-03-29T11:54:13.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.457+0000 7f6dfcff9640 1 -- 172.21.15.98:0/3258645663 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6de4004eb0 con 0x7f6df80d11e0 2024-03-29T11:54:13.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.457+0000 7f6dfcff9640 1 -- 172.21.15.98:0/3258645663 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 350241964 0 0) 0x7f6df8036110 con 0x7f6df80324b0 2024-03-29T11:54:13.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.457+0000 7f6dfcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:54:13.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.457+0000 7f6dfcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:54:13.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.457+0000 7f6dfcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:54:13.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.457+0000 7f6dfcff9640 10 monclient(hunting): my global_id is 4447 2024-03-29T11:54:13.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.457+0000 7f6dfcff9640 1 -- 172.21.15.98:0/3258645663 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6df80b8bf0 con 0x7f6df80324b0 2024-03-29T11:54:13.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.457+0000 7f6dfcff9640 1 -- 172.21.15.98:0/3258645663 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f6e0004fdd0 con 0x7f6df80d11e0 2024-03-29T11:54:13.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.457+0000 7f6dfcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:54:13.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.457+0000 7f6dfcff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:54:13.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.458+0000 7f6dfcff9640 10 monclient(hunting): dump: 2024-03-29T11:54:13.472 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:54:13.473 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:13.473 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:13.473 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:13.473 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:54:13.473 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:13.473 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:54:13.473 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:54:13.473 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:54:13.473 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:13.473 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.458+0000 7f6dfcff9640 1 -- 172.21.15.98:0/3258645663 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2148428632 0 0) 0x7f6df80363f0 con 0x7f6df8038c60 2024-03-29T11:54:13.473 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.458+0000 7f6dfcff9640 1 -- 172.21.15.98:0/3258645663 >> v1:172.21.15.98:6790/0 conn(0x7f6df80d11e0 legacy=0x7f6df80b84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:13.473 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.458+0000 7f6dfcff9640 1 -- 172.21.15.98:0/3258645663 >> v1:172.21.15.121:6789/0 conn(0x7f6df80324b0 legacy=0x7f6df802d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:13.473 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.458+0000 7f6dfcff9640 10 monclient: _finish_hunting 0 2024-03-29T11:54:13.473 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.458+0000 7f6dfcff9640 1 monclient: found mon.a 2024-03-29T11:54:13.473 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.458+0000 7f6dfcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:54:13.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.458+0000 7f6dfcff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:54:13.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.458+0000 7f6dfcff9640 1 -- 172.21.15.98:0/3258645663 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df80366d0 con 0x7f6df8038c60 2024-03-29T11:54:13.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.458+0000 7f6dfcff9640 10 monclient: _finish_auth 0 2024-03-29T11:54:13.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.458+0000 7f6dfcff9640 10 monclient: _check_auth_tickets 2024-03-29T11:54:13.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.458+0000 7f6dfcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:54:13.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.458+0000 7f6e052ba640 5 monclient: authenticate success, global_id 4425 2024-03-29T11:54:13.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.458+0000 7f6e052ba640 10 monclient: _renew_subs 2024-03-29T11:54:13.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.458+0000 7f6e052ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:54:13.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.458+0000 7f6e052ba640 1 -- 172.21.15.98:0/3258645663 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6df8034500 con 0x7f6df8038c60 2024-03-29T11:54:13.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.459+0000 7f6e052ba640 10 client.4425.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:54:13.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.459+0000 7f6e052ba640 10 monclient: _renew_subs 2024-03-29T11:54:13.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.459+0000 7f6e052ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:54:13.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.459+0000 7f6e052ba640 1 -- 172.21.15.98:0/3258645663 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6df8036ec0 con 0x7f6df8038c60 2024-03-29T11:54:13.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.459+0000 7f6e052ba640 1 librados: init done 2024-03-29T11:54:13.476 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.460+0000 7f6e052ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:54:13.476 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.461+0000 7f6e052ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:54:13.477 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.461+0000 7f6e052ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:54:13.477 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.461+0000 7f6e052ba640 1 -- 172.21.15.98:0/3258645663 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6df8007b90 con 0x7f6df8038c60 2024-03-29T11:54:13.477 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.461+0000 7f6dfcff9640 1 -- 172.21.15.98:0/3258645663 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6df4001cd0 con 0x7f6df8038c60 2024-03-29T11:54:13.477 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.461+0000 7f6dfcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:54:13.477 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.461+0000 7f6dfcff9640 1 -- 172.21.15.98:0/3258645663 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f6df4003a60 con 0x7f6df8038c60 2024-03-29T11:54:13.477 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.461+0000 7f6dfcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:54:13.477 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.461+0000 7f6dfcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:54:13.477 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.461+0000 7f6dfcff9640 10 monclient: dump: 2024-03-29T11:54:13.477 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:54:13.477 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:13.477 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:13.477 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:13.477 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:54:13.477 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:13.477 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:54:13.478 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:54:13.478 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:54:13.478 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:13.478 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.461+0000 7f6dfdffb640 4 set_mon_vals no callback set 2024-03-29T11:54:13.478 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.464+0000 7f6dfcff9640 1 -- 172.21.15.98:0/3258645663 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f6df4003c80 con 0x7f6df8038c60 2024-03-29T11:54:13.478 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.465+0000 7f6dfcff9640 1 -- 172.21.15.98:0/3258645663 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f6df4084b30 con 0x7f6df8038c60 2024-03-29T11:54:13.478 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.465+0000 7f6dfcff9640 10 client.4425.objecter ms_dispatch 0x7f6e00000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:54:13.478 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.465+0000 7f6dfcff9640 3 client.4425.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:54:13.478 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.465+0000 7f6dfcff9640 3 client.4425.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:54:13.478 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.466+0000 7f6dfcff9640 20 client.4425.objecter dump_active .. 0 homeless 2024-03-29T11:54:13.479 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.469+0000 7f6dfffff640 10 monclient: get_auth_request con 0x7f6de406db50 auth_method 0 2024-03-29T11:54:13.479 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.470+0000 7f6dfcff9640 10 client.4425.objecter ms_handle_connect 0x7f6de406db50 2024-03-29T11:54:13.479 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.470+0000 7f6dfcff9640 1 -- 172.21.15.98:0/3258645663 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6df4050410 con 0x7f6df8038c60 2024-03-29T11:54:13.479 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.470+0000 7f6dfcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:54:13.479 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.470+0000 7f6dfcff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:54:13.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.800+0000 7f6e052ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:54:13.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.800+0000 7f6e052ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:54:13.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.800+0000 7f6e052ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:54:13.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.800+0000 7f6e052ba640 1 -- 172.21.15.98:0/3258645663 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6df8034690 con 0x7f6df8038c60 2024-03-29T11:54:13.810 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.802+0000 7f6dfcff9640 1 -- 172.21.15.98:0/3258645663 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f6df4052340 con 0x7f6df8038c60 2024-03-29T11:54:13.810 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.802+0000 7f6dfcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:54:13.810 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.802+0000 7f6dfcff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:54:13.812 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:54:13.812 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:54:13.816 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.807+0000 7f6e052ba640 10 librados: watch_flush enter 2024-03-29T11:54:13.816 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.808+0000 7f6e052ba640 10 librados: watch_flush exit 2024-03-29T11:54:13.816 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.808+0000 7f6e052ba640 20 client.4425.objecter shutdown clearing up homeless session... 2024-03-29T11:54:13.816 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.808+0000 7f6e052ba640 10 client.4425.objecter successfully canceled tick 2024-03-29T11:54:13.819 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.810+0000 7f6e052ba640 1 -- 172.21.15.98:0/3258645663 >> v1:172.21.15.98:6816/4065186974 conn(0x7f6de406db50 legacy=0x7f6de4070010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:13.819 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.810+0000 7f6e052ba640 10 monclient: shutdown 2024-03-29T11:54:13.819 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.810+0000 7f6e052ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:54:13.819 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.810+0000 7f6e052ba640 1 -- 172.21.15.98:0/3258645663 >> v1:172.21.15.98:6789/0 conn(0x7f6df8038c60 legacy=0x7f6df80b7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:13.819 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.810+0000 7f6e052ba640 1 -- 172.21.15.98:0/3258645663 shutdown_connections 2024-03-29T11:54:13.819 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.810+0000 7f6e052ba640 1 -- 172.21.15.98:0/3258645663 >> 172.21.15.98:0/3258645663 conn(0x7f6df8005c20 msgr2=0x7f6df80c57d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:54:13.819 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.810+0000 7f6e052ba640 1 -- 172.21.15.98:0/3258645663 shutdown_connections 2024-03-29T11:54:13.819 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.810+0000 7f6e052ba640 1 -- 172.21.15.98:0/3258645663 wait complete. 2024-03-29T11:54:13.819 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:13.810+0000 7f6e052ba640 1 librados: shutdown 2024-03-29T11:54:13.835 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:54:20.642 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:54:20.791 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.782+0000 7fc8c0f15640 10 monclient: get_monmap_and_config 2024-03-29T11:54:20.791 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.782+0000 7fc8c0f15640 10 monclient: build_initial_monmap 2024-03-29T11:54:20.791 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.782+0000 7fc8c0f15640 10 monclient: monmap: 2024-03-29T11:54:20.791 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:54:20.791 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:20.791 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:54:20.783009+0000 2024-03-29T11:54:20.791 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:54:20.783009+0000 2024-03-29T11:54:20.791 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:54:20.791 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:20.791 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:54:20.791 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:54:20.791 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:54:20.792 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:20.794 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.786+0000 7fc8c0f15640 1 Processor -- start 2024-03-29T11:54:20.794 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.786+0000 7fc8c0f15640 1 -- start start 2024-03-29T11:54:20.794 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.786+0000 7fc8c0f15640 10 monclient: init 2024-03-29T11:54:20.795 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.787+0000 7fc8c0f15640 10 monclient: _reopen_session rank -1 2024-03-29T11:54:20.795 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.787+0000 7fc8c0f15640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:54:20.795 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.787+0000 7fc8c0f15640 10 monclient(hunting): picked mon.noname-a con 0x7fc8b4038a00 addr v1:172.21.15.98:6789/0 2024-03-29T11:54:20.795 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.788+0000 7fc8c0f15640 10 monclient(hunting): picked mon.noname-b con 0x7fc8b4038de0 addr v1:172.21.15.98:6790/0 2024-03-29T11:54:20.799 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.790+0000 7fc8c0f15640 10 monclient(hunting): picked mon.noname-c con 0x7fc8b4030680 addr v1:172.21.15.121:6789/0 2024-03-29T11:54:20.799 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.790+0000 7fc8ba59c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc8b4038a00 0x7fc8b4033540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:36279/0 (socket says 172.21.15.98:36279) 2024-03-29T11:54:20.799 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.790+0000 7fc8ba59c640 1 -- 172.21.15.98:0/1965056118 learned_addr learned my addr 172.21.15.98:0/1965056118 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:54:20.799 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.791+0000 7fc8c0f15640 1 -- 172.21.15.98:0/1965056118 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b40c54e0 con 0x7fc8b4038a00 2024-03-29T11:54:20.799 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.791+0000 7fc8c0f15640 1 -- 172.21.15.98:0/1965056118 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b4036450 con 0x7fc8b4030680 2024-03-29T11:54:20.799 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.791+0000 7fc8c0f15640 1 -- 172.21.15.98:0/1965056118 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b4039860 con 0x7fc8b4038de0 2024-03-29T11:54:20.799 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.791+0000 7fc8c0f15640 10 monclient(hunting): _renew_subs 2024-03-29T11:54:20.799 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.791+0000 7fc8c0f15640 10 monclient(hunting): authenticate will time out at 1167.876473s 2024-03-29T11:54:20.801 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.793+0000 7fc8b959a640 1 -- 172.21.15.98:0/1965056118 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 863209627 0 0) 0x7fc8b40c54e0 con 0x7fc8b4038a00 2024-03-29T11:54:20.803 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.794+0000 7fc8b959a640 10 monclient(hunting): _negotiate 2024-03-29T11:54:20.803 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.794+0000 7fc8b959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:54:20.803 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.794+0000 7fc8b959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:54:20.803 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.794+0000 7fc8b959a640 10 monclient(hunting): my global_id is 4428 2024-03-29T11:54:20.803 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.794+0000 7fc8b959a640 1 -- 172.21.15.98:0/1965056118 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8a4003520 con 0x7fc8b4038a00 2024-03-29T11:54:20.804 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.794+0000 7fc8b959a640 1 -- 172.21.15.98:0/1965056118 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1269327203 0 0) 0x7fc8b4036450 con 0x7fc8b4030680 2024-03-29T11:54:20.804 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.794+0000 7fc8b959a640 10 monclient(hunting): _negotiate 2024-03-29T11:54:20.804 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.794+0000 7fc8b959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:54:20.804 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.794+0000 7fc8b959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:54:20.804 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.794+0000 7fc8b959a640 10 monclient(hunting): my global_id is 4450 2024-03-29T11:54:20.804 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.794+0000 7fc8b959a640 1 -- 172.21.15.98:0/1965056118 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8b40c54e0 con 0x7fc8b4030680 2024-03-29T11:54:20.804 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.794+0000 7fc8b959a640 1 -- 172.21.15.98:0/1965056118 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2477265120 0 0) 0x7fc8b4039860 con 0x7fc8b4038de0 2024-03-29T11:54:20.804 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.794+0000 7fc8b959a640 10 monclient(hunting): _negotiate 2024-03-29T11:54:20.804 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.794+0000 7fc8b959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:54:20.804 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.794+0000 7fc8b959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:54:20.804 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.794+0000 7fc8b959a640 10 monclient(hunting): my global_id is 4460 2024-03-29T11:54:20.804 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.794+0000 7fc8b959a640 1 -- 172.21.15.98:0/1965056118 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8b4036450 con 0x7fc8b4038de0 2024-03-29T11:54:20.804 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.795+0000 7fc8b959a640 1 -- 172.21.15.98:0/1965056118 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3517629728 0 0) 0x7fc8a4003520 con 0x7fc8b4038a00 2024-03-29T11:54:20.804 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.795+0000 7fc8b959a640 1 -- 172.21.15.98:0/1965056118 >> v1:172.21.15.98:6790/0 conn(0x7fc8b4038de0 legacy=0x7fc8b40cadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:20.804 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.795+0000 7fc8b959a640 1 -- 172.21.15.98:0/1965056118 >> v1:172.21.15.121:6789/0 conn(0x7fc8b4030680 legacy=0x7fc8b40d3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:20.805 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.795+0000 7fc8b959a640 10 monclient: _finish_hunting 0 2024-03-29T11:54:20.805 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.795+0000 7fc8b959a640 1 monclient: found mon.noname-a 2024-03-29T11:54:20.805 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.795+0000 7fc8b959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:54:20.805 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.795+0000 7fc8b959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T11:54:20.805 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.795+0000 7fc8b959a640 1 -- 172.21.15.98:0/1965056118 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b40356b0 con 0x7fc8b4038a00 2024-03-29T11:54:20.805 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.795+0000 7fc8b959a640 10 monclient: _finish_auth 0 2024-03-29T11:54:20.805 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.795+0000 7fc8b959a640 10 monclient: _check_auth_tickets 2024-03-29T11:54:20.805 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.795+0000 7fc8b959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:54:20.805 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.795+0000 7fc8b959a640 1 -- 172.21.15.98:0/1965056118 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fc8b0001690 con 0x7fc8b4038a00 2024-03-29T11:54:20.805 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.795+0000 7fc8b959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:54:20.805 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.795+0000 7fc8b959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:54:20.805 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.795+0000 7fc8b959a640 10 monclient: dump: 2024-03-29T11:54:20.805 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:54:20.805 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:20.805 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:20.806 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:20.806 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:54:20.806 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:20.806 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:54:20.806 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:54:20.806 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:54:20.806 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:20.806 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.795+0000 7fc8c0f15640 5 monclient: authenticate success, global_id 4428 2024-03-29T11:54:20.806 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.795+0000 7fc8c0f15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:54:20.808 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.796+0000 7fc8b959a640 1 -- 172.21.15.98:0/1965056118 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8b00018b0 con 0x7fc8b4038a00 2024-03-29T11:54:20.808 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.796+0000 7fc8b959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:54:20.808 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.796+0000 7fc8b959a640 1 -- 172.21.15.98:0/1965056118 <== mon.0 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fc8b0001c40 con 0x7fc8b4038a00 2024-03-29T11:54:20.808 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.796+0000 7fc8c0f15640 10 monclient: get_monmap_and_config success 2024-03-29T11:54:20.808 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.796+0000 7fc8c0f15640 4 set_mon_vals no callback set 2024-03-29T11:54:20.808 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.796+0000 7fc8b959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:54:20.808 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.797+0000 7fc8b959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:54:20.808 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.797+0000 7fc8b959a640 10 monclient: dump: 2024-03-29T11:54:20.808 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:54:20.808 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:20.809 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:20.809 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:20.809 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:54:20.809 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:20.809 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:54:20.809 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:54:20.809 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:54:20.809 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:20.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.797+0000 7fc8c0f15640 10 monclient: shutdown 2024-03-29T11:54:20.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.797+0000 7fc8c0f15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:54:20.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.797+0000 7fc8c0f15640 1 -- 172.21.15.98:0/1965056118 >> v1:172.21.15.98:6789/0 conn(0x7fc8b4038a00 legacy=0x7fc8b4033540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:20.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.798+0000 7fc8c0f15640 1 -- 172.21.15.98:0/1965056118 shutdown_connections 2024-03-29T11:54:20.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.798+0000 7fc8c0f15640 1 -- 172.21.15.98:0/1965056118 >> 172.21.15.98:0/1965056118 conn(0x7fc8b4005c20 msgr2=0x7fc8b4006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:54:20.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.798+0000 7fc8c0f15640 1 -- 172.21.15.98:0/1965056118 shutdown_connections 2024-03-29T11:54:20.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.798+0000 7fc8c0f15640 1 -- 172.21.15.98:0/1965056118 wait complete. 2024-03-29T11:54:20.810 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.799+0000 7fc8c0f15640 10 monclient: build_initial_monmap 2024-03-29T11:54:20.810 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.799+0000 7fc8c0f15640 10 monclient: monmap: 2024-03-29T11:54:20.810 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:54:20.810 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:54:20.810 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:54:20.810 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:54:20.810 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:54:20.810 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:20.810 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:54:20.810 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:54:20.810 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:54:20.810 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:20.810 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.800+0000 7fc8c0f15640 1 librados: starting msgr at 2024-03-29T11:54:20.810 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.800+0000 7fc8c0f15640 1 librados: starting objecter 2024-03-29T11:54:20.810 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.800+0000 7fc8c0f15640 1 Processor -- start 2024-03-29T11:54:20.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.801+0000 7fc8c0f15640 1 -- start start 2024-03-29T11:54:20.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.801+0000 7fc8c0f15640 1 librados: setting wanted keys 2024-03-29T11:54:20.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.801+0000 7fc8c0f15640 1 librados: calling monclient init 2024-03-29T11:54:20.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.801+0000 7fc8c0f15640 10 monclient: init 2024-03-29T11:54:20.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.802+0000 7fc8c0f15640 10 monclient: _reopen_session rank -1 2024-03-29T11:54:20.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.803+0000 7fc8c0f15640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:54:20.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.803+0000 7fc8c0f15640 10 monclient(hunting): picked mon.noname-c con 0x7fc8b4030680 addr v1:172.21.15.98:6790/0 2024-03-29T11:54:20.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.803+0000 7fc8c0f15640 10 monclient(hunting): picked mon.noname-a con 0x7fc8b4038a00 addr v1:172.21.15.98:6789/0 2024-03-29T11:54:20.813 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.805+0000 7fc8c0f15640 10 monclient(hunting): picked mon.noname-b con 0x7fc8b402f360 addr v1:172.21.15.121:6789/0 2024-03-29T11:54:20.819 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.805+0000 7fc8c0f15640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b40c0c80 con 0x7fc8b4038a00 2024-03-29T11:54:20.819 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.805+0000 7fc8c0f15640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b40ca730 con 0x7fc8b402f360 2024-03-29T11:54:20.819 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.805+0000 7fc8c0f15640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b40ca420 con 0x7fc8b4030680 2024-03-29T11:54:20.819 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.805+0000 7fc8c0f15640 10 monclient(hunting): _renew_subs 2024-03-29T11:54:20.819 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.805+0000 7fc8c0f15640 10 monclient(hunting): authenticate will time out at 1167.890124s 2024-03-29T11:54:20.819 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.805+0000 7fc8ba59c640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fc8b4030680 0x7fc8b415c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:37274/0 (socket says 172.21.15.98:37274) 2024-03-29T11:54:20.819 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.805+0000 7fc8ba59c640 1 -- 172.21.15.98:0/3471746642 learned_addr learned my addr 172.21.15.98:0/3471746642 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:54:20.819 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.806+0000 7fc8aaffd640 10 client.?.objecter ms_handle_connect 0x7fc8b4030680 2024-03-29T11:54:20.819 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.806+0000 7fc8aaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:54:20.819 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.806+0000 7fc8aaffd640 1 -- 172.21.15.98:0/3471746642 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 323002338 0 0) 0x7fc8b40ca420 con 0x7fc8b4030680 2024-03-29T11:54:20.820 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.806+0000 7fc8aaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:54:20.820 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.806+0000 7fc8aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:54:20.820 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.806+0000 7fc8aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:54:20.820 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.806+0000 7fc8aaffd640 10 monclient(hunting): my global_id is 4463 2024-03-29T11:54:20.820 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.806+0000 7fc8aaffd640 1 -- 172.21.15.98:0/3471746642 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc89c004eb0 con 0x7fc8b4030680 2024-03-29T11:54:20.820 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.809+0000 7fc8aaffd640 10 client.?.objecter ms_handle_connect 0x7fc8b4038a00 2024-03-29T11:54:20.820 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.809+0000 7fc8aaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:54:20.820 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.809+0000 7fc8aaffd640 1 -- 172.21.15.98:0/3471746642 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3582819750 0 0) 0x7fc89c004eb0 con 0x7fc8b4030680 2024-03-29T11:54:20.820 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.809+0000 7fc8aaffd640 1 -- 172.21.15.98:0/3471746642 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8b40ca420 con 0x7fc8b4030680 2024-03-29T11:54:20.820 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.809+0000 7fc8aaffd640 1 -- 172.21.15.98:0/3471746642 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fc8b00047e0 con 0x7fc8b4030680 2024-03-29T11:54:20.820 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.809+0000 7fc8aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:54:20.820 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.809+0000 7fc8aaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:54:20.820 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.809+0000 7fc8aaffd640 10 monclient(hunting): dump: 2024-03-29T11:54:20.820 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:54:20.820 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:20.821 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:20.821 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:20.821 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:54:20.821 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:20.821 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:54:20.821 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:54:20.821 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:54:20.821 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:20.821 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.809+0000 7fc8aaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:54:20.821 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.810+0000 7fc8aaffd640 1 -- 172.21.15.98:0/3471746642 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 901522866 0 0) 0x7fc8b40ca420 con 0x7fc8b4030680 2024-03-29T11:54:20.821 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.810+0000 7fc8aaffd640 1 -- 172.21.15.98:0/3471746642 >> v1:172.21.15.121:6789/0 conn(0x7fc8b402f360 legacy=0x7fc8b40c0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:20.821 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.810+0000 7fc8aaffd640 1 -- 172.21.15.98:0/3471746642 >> v1:172.21.15.98:6789/0 conn(0x7fc8b4038a00 legacy=0x7fc8b40bfe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:20.821 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.810+0000 7fc8aaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:54:20.821 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.810+0000 7fc8aaffd640 1 monclient: found mon.c 2024-03-29T11:54:20.821 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.810+0000 7fc8aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:54:20.822 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.810+0000 7fc8aaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:54:20.822 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.810+0000 7fc8aaffd640 1 -- 172.21.15.98:0/3471746642 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b402fd70 con 0x7fc8b4030680 2024-03-29T11:54:20.822 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.810+0000 7fc8aaffd640 10 monclient: _finish_auth 0 2024-03-29T11:54:20.822 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.810+0000 7fc8aaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:54:20.822 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.810+0000 7fc8aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:54:20.822 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.810+0000 7fc8c0f15640 5 monclient: authenticate success, global_id 4463 2024-03-29T11:54:20.822 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.810+0000 7fc8c0f15640 10 monclient: _renew_subs 2024-03-29T11:54:20.822 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.810+0000 7fc8c0f15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:54:20.822 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.810+0000 7fc8c0f15640 1 -- 172.21.15.98:0/3471746642 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8b402ffc0 con 0x7fc8b4030680 2024-03-29T11:54:20.822 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.810+0000 7fc8c0f15640 10 client.4463.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:54:20.822 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.810+0000 7fc8c0f15640 10 monclient: _renew_subs 2024-03-29T11:54:20.822 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.810+0000 7fc8c0f15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:54:20.822 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.810+0000 7fc8c0f15640 1 -- 172.21.15.98:0/3471746642 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8b40c5ac0 con 0x7fc8b4030680 2024-03-29T11:54:20.822 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.810+0000 7fc8c0f15640 1 librados: init done 2024-03-29T11:54:20.823 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.811+0000 7fc8aaffd640 10 client.4463.objecter ms_handle_connect 0x7fc8b402f360 2024-03-29T11:54:20.823 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.811+0000 7fc8aaffd640 10 client.4463.objecter resend_mon_ops 2024-03-29T11:54:20.824 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.814+0000 7fc8aaffd640 1 -- 172.21.15.98:0/3471746642 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8b0004bf0 con 0x7fc8b4030680 2024-03-29T11:54:20.824 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.814+0000 7fc8aaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:54:20.824 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.814+0000 7fc8aaffd640 1 -- 172.21.15.98:0/3471746642 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fc8b0005000 con 0x7fc8b4030680 2024-03-29T11:54:20.824 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.814+0000 7fc8aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:54:20.824 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.814+0000 7fc8aaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:54:20.824 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.814+0000 7fc8b8d99640 4 set_mon_vals no callback set 2024-03-29T11:54:20.824 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.815+0000 7fc8aaffd640 10 monclient: dump: 2024-03-29T11:54:20.824 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:54:20.824 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:20.824 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:20.824 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:20.824 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:54:20.824 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:20.824 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:54:20.824 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:54:20.825 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:54:20.825 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:20.825 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.815+0000 7fc8aaffd640 1 -- 172.21.15.98:0/3471746642 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7fc8b0005220 con 0x7fc8b4030680 2024-03-29T11:54:20.825 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.815+0000 7fc8c0f15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:54:20.825 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.815+0000 7fc8c0f15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:54:20.825 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.815+0000 7fc8c0f15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:54:20.825 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.815+0000 7fc8c0f15640 1 -- 172.21.15.98:0/3471746642 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8a0005080 con 0x7fc8b4030680 2024-03-29T11:54:20.825 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.816+0000 7fc8b9d9b640 10 monclient: get_auth_request con 0x7fc89c06dc40 auth_method 0 2024-03-29T11:54:20.826 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.816+0000 7fc8aaffd640 10 client.4463.objecter ms_handle_connect 0x7fc89c06dc40 2024-03-29T11:54:20.826 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.817+0000 7fc8aaffd640 1 -- 172.21.15.98:0/3471746642 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7fc8b0084d30 con 0x7fc8b4030680 2024-03-29T11:54:20.826 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.817+0000 7fc8aaffd640 10 client.4463.objecter ms_dispatch 0x7fc8bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:54:20.826 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.817+0000 7fc8aaffd640 3 client.4463.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:54:20.826 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.817+0000 7fc8aaffd640 3 client.4463.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:54:20.826 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.817+0000 7fc8aaffd640 20 client.4463.objecter dump_active .. 0 homeless 2024-03-29T11:54:20.836 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.827+0000 7fc8aaffd640 1 -- 172.21.15.98:0/3471746642 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc8b0050610 con 0x7fc8b4030680 2024-03-29T11:54:20.836 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.827+0000 7fc8aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:54:20.836 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:20.827+0000 7fc8aaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:54:21.156 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:21.148+0000 7fc8c0f15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:54:21.156 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:21.148+0000 7fc8c0f15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:54:21.156 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:21.148+0000 7fc8c0f15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:54:21.157 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:21.148+0000 7fc8c0f15640 1 -- 172.21.15.98:0/3471746642 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc8a0005330 con 0x7fc8b4030680 2024-03-29T11:54:21.157 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:21.149+0000 7fc8aaffd640 1 -- 172.21.15.98:0/3471746642 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7fc8b0052540 con 0x7fc8b4030680 2024-03-29T11:54:21.157 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:21.149+0000 7fc8aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:54:21.157 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:21.149+0000 7fc8aaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:54:21.160 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:54:21.160 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:54:21.161 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:21.153+0000 7fc8c0f15640 10 librados: watch_flush enter 2024-03-29T11:54:21.161 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:21.153+0000 7fc8c0f15640 10 librados: watch_flush exit 2024-03-29T11:54:21.161 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:21.153+0000 7fc8c0f15640 20 client.4463.objecter shutdown clearing up homeless session... 2024-03-29T11:54:21.161 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:21.153+0000 7fc8c0f15640 10 client.4463.objecter successfully canceled tick 2024-03-29T11:54:21.162 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:21.153+0000 7fc8c0f15640 1 -- 172.21.15.98:0/3471746642 >> v1:172.21.15.98:6816/4065186974 conn(0x7fc89c06dc40 legacy=0x7fc89c070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:21.162 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:21.153+0000 7fc8c0f15640 10 monclient: shutdown 2024-03-29T11:54:21.162 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:21.154+0000 7fc8c0f15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:54:21.162 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:21.154+0000 7fc8c0f15640 1 -- 172.21.15.98:0/3471746642 >> v1:172.21.15.98:6790/0 conn(0x7fc8b4030680 legacy=0x7fc8b415c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:21.163 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:21.154+0000 7fc8c0f15640 1 -- 172.21.15.98:0/3471746642 shutdown_connections 2024-03-29T11:54:21.163 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:21.154+0000 7fc8c0f15640 1 -- 172.21.15.98:0/3471746642 >> 172.21.15.98:0/3471746642 conn(0x7fc8b4005c20 msgr2=0x7fc8b403ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:54:21.163 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:21.154+0000 7fc8c0f15640 1 -- 172.21.15.98:0/3471746642 shutdown_connections 2024-03-29T11:54:21.163 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:21.155+0000 7fc8c0f15640 1 -- 172.21.15.98:0/3471746642 wait complete. 2024-03-29T11:54:21.163 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:21.155+0000 7fc8c0f15640 1 librados: shutdown 2024-03-29T11:54:21.179 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:54:27.986 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:54:28.137 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.129+0000 7f07549e6640 10 monclient: get_monmap_and_config 2024-03-29T11:54:28.137 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.129+0000 7f07549e6640 10 monclient: build_initial_monmap 2024-03-29T11:54:28.137 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.129+0000 7f07549e6640 10 monclient: monmap: 2024-03-29T11:54:28.137 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:54:28.137 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:28.137 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:54:28.130048+0000 2024-03-29T11:54:28.137 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:54:28.130048+0000 2024-03-29T11:54:28.137 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:54:28.137 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:28.137 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:54:28.138 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:54:28.138 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:54:28.138 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:28.138 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.130+0000 7f07549e6640 1 Processor -- start 2024-03-29T11:54:28.138 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.130+0000 7f07549e6640 1 -- start start 2024-03-29T11:54:28.138 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.130+0000 7f07549e6640 10 monclient: init 2024-03-29T11:54:28.139 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.131+0000 7f07549e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:54:28.141 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.131+0000 7f07549e6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:54:28.141 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.131+0000 7f07549e6640 10 monclient(hunting): picked mon.noname-a con 0x7f07500b1f60 addr v1:172.21.15.98:6789/0 2024-03-29T11:54:28.141 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.131+0000 7f07549e6640 10 monclient(hunting): picked mon.noname-b con 0x7f0750071eb0 addr v1:172.21.15.98:6790/0 2024-03-29T11:54:28.141 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.131+0000 7f07549e6640 10 monclient(hunting): picked mon.noname-c con 0x7f0750143d10 addr v1:172.21.15.121:6789/0 2024-03-29T11:54:28.142 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.131+0000 7f07549e6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f075006ef80 con 0x7f07500b1f60 2024-03-29T11:54:28.142 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.131+0000 7f07549e6640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f075006f260 con 0x7f0750143d10 2024-03-29T11:54:28.142 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.131+0000 7f07549e6640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07500b40c0 con 0x7f0750071eb0 2024-03-29T11:54:28.142 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.131+0000 7f07549e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:54:28.142 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.131+0000 7f07549e6640 10 monclient(hunting): authenticate will time out at 1175.216714s 2024-03-29T11:54:28.142 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.134+0000 7f074dd9b640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f0750071eb0 0x7f0750146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:53404/0 (socket says 172.21.15.98:53404) 2024-03-29T11:54:28.142 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.134+0000 7f074dd9b640 1 -- 172.21.15.98:0/2278451745 learned_addr learned my addr 172.21.15.98:0/2278451745 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:54:28.143 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.135+0000 7f074d59a640 1 -- 172.21.15.98:0/2278451745 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1762896145 0 0) 0x7f075006f260 con 0x7f0750143d10 2024-03-29T11:54:28.143 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.135+0000 7f074d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:54:28.143 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.135+0000 7f074d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:54:28.143 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.135+0000 7f074d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:54:28.143 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.135+0000 7f074d59a640 10 monclient(hunting): my global_id is 4456 2024-03-29T11:54:28.143 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.135+0000 7f074d59a640 1 -- 172.21.15.98:0/2278451745 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0738003520 con 0x7f0750143d10 2024-03-29T11:54:28.144 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.135+0000 7f074d59a640 1 -- 172.21.15.98:0/2278451745 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 230402666 0 0) 0x7f07500b40c0 con 0x7f0750071eb0 2024-03-29T11:54:28.144 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.135+0000 7f074d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:54:28.144 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.135+0000 7f074d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:54:28.144 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.135+0000 7f074d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:54:28.144 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.135+0000 7f074d59a640 10 monclient(hunting): my global_id is 4466 2024-03-29T11:54:28.144 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.135+0000 7f074d59a640 1 -- 172.21.15.98:0/2278451745 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07380048a0 con 0x7f0750071eb0 2024-03-29T11:54:28.144 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.136+0000 7f074d59a640 1 -- 172.21.15.98:0/2278451745 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1047299173 0 0) 0x7f07380048a0 con 0x7f0750071eb0 2024-03-29T11:54:28.144 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.136+0000 7f074d59a640 1 -- 172.21.15.98:0/2278451745 >> v1:172.21.15.121:6789/0 conn(0x7f0750143d10 legacy=0x7f0750146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:28.144 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.136+0000 7f074d59a640 1 -- 172.21.15.98:0/2278451745 >> v1:172.21.15.98:6789/0 conn(0x7f07500b1f60 legacy=0x7f0750143600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:54:28.146 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.137+0000 7f074d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:54:28.146 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.137+0000 7f074d59a640 1 monclient: found mon.noname-b 2024-03-29T11:54:28.146 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.137+0000 7f074d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:54:28.146 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.137+0000 7f074d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:54:28.146 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.137+0000 7f074d59a640 1 -- 172.21.15.98:0/2278451745 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0750155d70 con 0x7f0750071eb0 2024-03-29T11:54:28.146 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.137+0000 7f074d59a640 10 monclient: _finish_auth 0 2024-03-29T11:54:28.146 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.137+0000 7f074d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:54:28.146 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.137+0000 7f074d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:54:28.148 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.137+0000 7f074d59a640 1 -- 172.21.15.98:0/2278451745 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f0744001630 con 0x7f0750071eb0 2024-03-29T11:54:28.148 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.137+0000 7f074d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:54:28.149 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.137+0000 7f074d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:54:28.149 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.137+0000 7f074d59a640 10 monclient: dump: 2024-03-29T11:54:28.149 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:54:28.149 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:28.149 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:28.149 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:28.149 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:54:28.149 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:28.149 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:54:28.149 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:54:28.149 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:54:28.149 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:28.149 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.137+0000 7f07549e6640 5 monclient: authenticate success, global_id 4466 2024-03-29T11:54:28.149 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.137+0000 7f07549e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:54:28.149 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.138+0000 7f074d59a640 1 -- 172.21.15.98:0/2278451745 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0744001880 con 0x7f0750071eb0 2024-03-29T11:54:28.150 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.138+0000 7f074d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:54:28.150 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.138+0000 7f07549e6640 10 monclient: get_monmap_and_config success 2024-03-29T11:54:28.150 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.138+0000 7f07549e6640 4 set_mon_vals no callback set 2024-03-29T11:54:28.150 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.139+0000 7f07549e6640 10 monclient: shutdown 2024-03-29T11:54:28.150 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.139+0000 7f07549e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:54:28.150 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.139+0000 7f07549e6640 1 -- 172.21.15.98:0/2278451745 >> v1:172.21.15.98:6790/0 conn(0x7f0750071eb0 legacy=0x7f0750146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:28.150 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.139+0000 7f074d59a640 1 -- 172.21.15.98:0/2278451745 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f0744001c10 con 0x7f0750071eb0 2024-03-29T11:54:28.150 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.139+0000 7f074d59a640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:54:28.150 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.139+0000 7f07549e6640 1 -- 172.21.15.98:0/2278451745 shutdown_connections 2024-03-29T11:54:28.150 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.139+0000 7f07549e6640 1 -- 172.21.15.98:0/2278451745 >> 172.21.15.98:0/2278451745 conn(0x7f0750066a00 msgr2=0x7f0750066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:54:28.150 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.139+0000 7f07549e6640 1 -- 172.21.15.98:0/2278451745 shutdown_connections 2024-03-29T11:54:28.150 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.139+0000 7f07549e6640 1 -- 172.21.15.98:0/2278451745 wait complete. 2024-03-29T11:54:28.151 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.141+0000 7f07549e6640 10 monclient: build_initial_monmap 2024-03-29T11:54:28.151 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.141+0000 7f07549e6640 10 monclient: monmap: 2024-03-29T11:54:28.152 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:54:28.152 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:54:28.152 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:54:28.152 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:54:28.152 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:54:28.152 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:28.152 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:54:28.152 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:54:28.152 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:54:28.152 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:28.152 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.141+0000 7f07549e6640 1 librados: starting msgr at 2024-03-29T11:54:28.152 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.141+0000 7f07549e6640 1 librados: starting objecter 2024-03-29T11:54:28.152 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.141+0000 7f07549e6640 1 Processor -- start 2024-03-29T11:54:28.152 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.141+0000 7f07549e6640 1 -- start start 2024-03-29T11:54:28.153 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.141+0000 7f07549e6640 1 librados: setting wanted keys 2024-03-29T11:54:28.153 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.141+0000 7f07549e6640 1 librados: calling monclient init 2024-03-29T11:54:28.153 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.141+0000 7f07549e6640 10 monclient: init 2024-03-29T11:54:28.153 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.142+0000 7f07549e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:54:28.153 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.142+0000 7f07549e6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:54:28.153 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.143+0000 7f07549e6640 10 monclient(hunting): picked mon.noname-b con 0x7f07500b1f60 addr v1:172.21.15.121:6789/0 2024-03-29T11:54:28.153 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.143+0000 7f07549e6640 10 monclient(hunting): picked mon.noname-c con 0x7f0750143d10 addr v1:172.21.15.98:6790/0 2024-03-29T11:54:28.153 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.143+0000 7f07549e6640 10 monclient(hunting): picked mon.noname-a con 0x7f07501491f0 addr v1:172.21.15.98:6789/0 2024-03-29T11:54:28.153 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.143+0000 7f07549e6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07501496e0 con 0x7f07501491f0 2024-03-29T11:54:28.153 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.143+0000 7f07549e6640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07501499c0 con 0x7f07500b1f60 2024-03-29T11:54:28.153 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.143+0000 7f07549e6640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f075014b130 con 0x7f0750143d10 2024-03-29T11:54:28.153 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.143+0000 7f07549e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:54:28.153 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.143+0000 7f07549e6640 10 monclient(hunting): authenticate will time out at 1175.228414s 2024-03-29T11:54:28.154 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.145+0000 7f074e59c640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f07500b1f60 0x7f0750149e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:54709/0 (socket says 172.21.15.98:54709) 2024-03-29T11:54:28.154 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.145+0000 7f074e59c640 1 -- 172.21.15.98:0/2857243034 learned_addr learned my addr 172.21.15.98:0/2857243034 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:54:28.154 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.146+0000 7f073effd640 10 client.?.objecter ms_handle_connect 0x7f07500b1f60 2024-03-29T11:54:28.154 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.146+0000 7f073effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:54:28.154 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.146+0000 7f073effd640 1 -- 172.21.15.98:0/2857243034 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2197826565 0 0) 0x7f07501499c0 con 0x7f07500b1f60 2024-03-29T11:54:28.154 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.146+0000 7f073effd640 10 monclient(hunting): _negotiate 2024-03-29T11:54:28.155 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.146+0000 7f073effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:54:28.155 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.146+0000 7f073effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:54:28.155 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.146+0000 7f073effd640 10 monclient(hunting): my global_id is 4459 2024-03-29T11:54:28.155 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.146+0000 7f073effd640 1 -- 172.21.15.98:0/2857243034 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0730004eb0 con 0x7f07500b1f60 2024-03-29T11:54:28.158 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.149+0000 7f073effd640 10 client.?.objecter ms_handle_connect 0x7f0750143d10 2024-03-29T11:54:28.158 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.149+0000 7f073effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:54:28.158 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.149+0000 7f073effd640 1 -- 172.21.15.98:0/2857243034 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1688005925 0 0) 0x7f0730004eb0 con 0x7f07500b1f60 2024-03-29T11:54:28.158 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.149+0000 7f073effd640 1 -- 172.21.15.98:0/2857243034 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f07501499c0 con 0x7f07500b1f60 2024-03-29T11:54:28.158 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.149+0000 7f073effd640 1 -- 172.21.15.98:0/2857243034 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3561646466 0 0) 0x7f075014b130 con 0x7f0750143d10 2024-03-29T11:54:28.158 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.149+0000 7f073effd640 10 monclient(hunting): _negotiate 2024-03-29T11:54:28.158 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.149+0000 7f073effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:54:28.158 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.149+0000 7f073effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:54:28.158 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.149+0000 7f073effd640 10 monclient(hunting): my global_id is 4469 2024-03-29T11:54:28.159 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.149+0000 7f073effd640 1 -- 172.21.15.98:0/2857243034 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0730004eb0 con 0x7f0750143d10 2024-03-29T11:54:28.159 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.150+0000 7f073effd640 1 -- 172.21.15.98:0/2857243034 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f0740001960 con 0x7f07500b1f60 2024-03-29T11:54:28.159 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.150+0000 7f073effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:54:28.159 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.150+0000 7f073effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:54:28.159 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.150+0000 7f073effd640 10 monclient(hunting): dump: 2024-03-29T11:54:28.159 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:54:28.159 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:28.159 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:28.159 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:28.159 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:54:28.159 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:28.159 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:54:28.159 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:54:28.159 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:54:28.159 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:28.160 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.150+0000 7f073effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:54:28.162 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.153+0000 7f073effd640 1 -- 172.21.15.98:0/2857243034 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3386734550 0 0) 0x7f07501499c0 con 0x7f07500b1f60 2024-03-29T11:54:28.162 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.153+0000 7f073effd640 1 -- 172.21.15.98:0/2857243034 >> v1:172.21.15.98:6790/0 conn(0x7f0750143d10 legacy=0x7f0750147790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:28.163 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.153+0000 7f073effd640 1 -- 172.21.15.98:0/2857243034 >> v1:172.21.15.98:6789/0 conn(0x7f07501491f0 legacy=0x7f07501e60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:28.163 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.153+0000 7f073effd640 10 monclient: _finish_hunting 0 2024-03-29T11:54:28.163 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.153+0000 7f073effd640 1 monclient: found mon.b 2024-03-29T11:54:28.163 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.153+0000 7f073effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:54:28.163 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.153+0000 7f073effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:54:28.163 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.153+0000 7f073effd640 1 -- 172.21.15.98:0/2857243034 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f075014b410 con 0x7f07500b1f60 2024-03-29T11:54:28.163 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.154+0000 7f073effd640 10 monclient: _finish_auth 0 2024-03-29T11:54:28.163 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.154+0000 7f073effd640 10 monclient: _check_auth_tickets 2024-03-29T11:54:28.163 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.154+0000 7f073effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:54:28.163 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.154+0000 7f073effd640 10 client.?.objecter ms_handle_connect 0x7f07501491f0 2024-03-29T11:54:28.163 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.154+0000 7f07549e6640 5 monclient: authenticate success, global_id 4459 2024-03-29T11:54:28.163 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.154+0000 7f07549e6640 10 monclient: _renew_subs 2024-03-29T11:54:28.163 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.154+0000 7f07549e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:54:28.163 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.154+0000 7f07549e6640 1 -- 172.21.15.98:0/2857243034 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0750070920 con 0x7f07500b1f60 2024-03-29T11:54:28.163 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.154+0000 7f07549e6640 10 client.4459.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:54:28.164 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.154+0000 7f07549e6640 10 monclient: _renew_subs 2024-03-29T11:54:28.164 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.154+0000 7f07549e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:54:28.164 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.154+0000 7f07549e6640 1 -- 172.21.15.98:0/2857243034 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f07501e77d0 con 0x7f07500b1f60 2024-03-29T11:54:28.164 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.154+0000 7f07549e6640 1 librados: init done 2024-03-29T11:54:28.164 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.154+0000 7f073effd640 10 client.4459.objecter resend_mon_ops 2024-03-29T11:54:28.165 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.155+0000 7f073effd640 1 -- 172.21.15.98:0/2857243034 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0740001d00 con 0x7f07500b1f60 2024-03-29T11:54:28.165 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.155+0000 7f073effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:54:28.165 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.155+0000 7f073effd640 1 -- 172.21.15.98:0/2857243034 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f0740001b80 con 0x7f07500b1f60 2024-03-29T11:54:28.165 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.155+0000 7f073effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:54:28.165 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.155+0000 7f073effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:54:28.165 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.155+0000 7f073effd640 10 monclient: dump: 2024-03-29T11:54:28.165 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:54:28.165 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:28.165 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:28.165 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:28.166 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:54:28.166 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:28.166 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:54:28.166 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:54:28.166 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:54:28.166 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:28.166 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.155+0000 7f074cd99640 4 set_mon_vals no callback set 2024-03-29T11:54:28.166 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.156+0000 7f07549e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:54:28.166 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.156+0000 7f07549e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:54:28.166 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.156+0000 7f07549e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:54:28.166 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.156+0000 7f07549e6640 1 -- 172.21.15.98:0/2857243034 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0734005080 con 0x7f07500b1f60 2024-03-29T11:54:28.167 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.159+0000 7f073effd640 1 -- 172.21.15.98:0/2857243034 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f0740003a40 con 0x7f07500b1f60 2024-03-29T11:54:28.168 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.160+0000 7f073effd640 1 -- 172.21.15.98:0/2857243034 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f0740084aa0 con 0x7f07500b1f60 2024-03-29T11:54:28.168 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.160+0000 7f074dd9b640 10 monclient: get_auth_request con 0x7f073006dbf0 auth_method 0 2024-03-29T11:54:28.168 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.160+0000 7f073effd640 10 client.4459.objecter ms_dispatch 0x7f0750000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:54:28.168 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.160+0000 7f073effd640 3 client.4459.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:54:28.169 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.160+0000 7f073effd640 3 client.4459.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:54:28.169 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.161+0000 7f073effd640 20 client.4459.objecter dump_active .. 0 homeless 2024-03-29T11:54:28.169 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.161+0000 7f073effd640 10 client.4459.objecter ms_handle_connect 0x7f073006dbf0 2024-03-29T11:54:28.176 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.166+0000 7f073effd640 1 -- 172.21.15.98:0/2857243034 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f07400501b0 con 0x7f07500b1f60 2024-03-29T11:54:28.176 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.166+0000 7f073effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:54:28.176 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.166+0000 7f073effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:54:28.497 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.489+0000 7f07549e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:54:28.497 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.489+0000 7f07549e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:54:28.497 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.489+0000 7f07549e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:54:28.497 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.489+0000 7f07549e6640 1 -- 172.21.15.98:0/2857243034 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0734005330 con 0x7f07500b1f60 2024-03-29T11:54:28.498 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.490+0000 7f073effd640 1 -- 172.21.15.98:0/2857243034 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f07400522b0 con 0x7f07500b1f60 2024-03-29T11:54:28.498 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.490+0000 7f073effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:54:28.498 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.490+0000 7f073effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:54:28.498 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:54:28.499 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:54:28.502 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.494+0000 7f07549e6640 10 librados: watch_flush enter 2024-03-29T11:54:28.502 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.494+0000 7f07549e6640 10 librados: watch_flush exit 2024-03-29T11:54:28.503 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.494+0000 7f07549e6640 20 client.4459.objecter shutdown clearing up homeless session... 2024-03-29T11:54:28.503 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.494+0000 7f07549e6640 10 client.4459.objecter successfully canceled tick 2024-03-29T11:54:28.503 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.495+0000 7f07549e6640 1 -- 172.21.15.98:0/2857243034 >> v1:172.21.15.98:6816/4065186974 conn(0x7f073006dbf0 legacy=0x7f07300700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:28.503 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.495+0000 7f07549e6640 10 monclient: shutdown 2024-03-29T11:54:28.503 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.495+0000 7f07549e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:54:28.503 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.495+0000 7f07549e6640 1 -- 172.21.15.98:0/2857243034 >> v1:172.21.15.121:6789/0 conn(0x7f07500b1f60 legacy=0x7f0750149e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:28.503 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.495+0000 7f07549e6640 1 -- 172.21.15.98:0/2857243034 shutdown_connections 2024-03-29T11:54:28.503 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.495+0000 7f07549e6640 1 -- 172.21.15.98:0/2857243034 >> 172.21.15.98:0/2857243034 conn(0x7f0750066a00 msgr2=0x7f0750142ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:54:28.505 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.497+0000 7f07549e6640 1 -- 172.21.15.98:0/2857243034 shutdown_connections 2024-03-29T11:54:28.505 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.497+0000 7f07549e6640 1 -- 172.21.15.98:0/2857243034 wait complete. 2024-03-29T11:54:28.505 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:28.497+0000 7f07549e6640 1 librados: shutdown 2024-03-29T11:54:28.523 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:54:35.330 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:54:35.477 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.469+0000 7f78b1a81640 10 monclient: get_monmap_and_config 2024-03-29T11:54:35.477 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.469+0000 7f78b1a81640 10 monclient: build_initial_monmap 2024-03-29T11:54:35.477 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.469+0000 7f78b1a81640 10 monclient: monmap: 2024-03-29T11:54:35.477 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:54:35.477 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:35.478 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:54:35.470527+0000 2024-03-29T11:54:35.478 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:54:35.470527+0000 2024-03-29T11:54:35.478 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:54:35.478 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:35.478 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:54:35.478 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:54:35.478 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:54:35.478 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:35.479 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.470+0000 7f78b1a81640 1 Processor -- start 2024-03-29T11:54:35.479 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.470+0000 7f78b1a81640 1 -- start start 2024-03-29T11:54:35.479 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.470+0000 7f78b1a81640 10 monclient: init 2024-03-29T11:54:35.479 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.471+0000 7f78b1a81640 10 monclient: _reopen_session rank -1 2024-03-29T11:54:35.479 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.471+0000 7f78b1a81640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:54:35.479 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.471+0000 7f78b1a81640 10 monclient(hunting): picked mon.noname-c con 0x7f78ac1457f0 addr v1:172.21.15.121:6789/0 2024-03-29T11:54:35.479 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.471+0000 7f78b1a81640 10 monclient(hunting): picked mon.noname-b con 0x7f78ac13f650 addr v1:172.21.15.98:6790/0 2024-03-29T11:54:35.480 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.472+0000 7f78aaffd640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f78ac13f650 0x7f78ac0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:50336/0 (socket says 172.21.15.98:50336) 2024-03-29T11:54:35.481 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.472+0000 7f78b1a81640 10 monclient(hunting): picked mon.noname-a con 0x7f78ac0a9d70 addr v1:172.21.15.98:6789/0 2024-03-29T11:54:35.481 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.472+0000 7f78aaffd640 1 -- 172.21.15.98:0/2149168557 learned_addr learned my addr 172.21.15.98:0/2149168557 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:54:35.481 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.472+0000 7f78b1a81640 1 -- 172.21.15.98:0/2149168557 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78ac0aaef0 con 0x7f78ac0a9d70 2024-03-29T11:54:35.481 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.472+0000 7f78b1a81640 1 -- 172.21.15.98:0/2149168557 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78ac145bd0 con 0x7f78ac1457f0 2024-03-29T11:54:35.481 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.472+0000 7f78b1a81640 1 -- 172.21.15.98:0/2149168557 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78ac145eb0 con 0x7f78ac13f650 2024-03-29T11:54:35.481 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.472+0000 7f78b1a81640 10 monclient(hunting): _renew_subs 2024-03-29T11:54:35.481 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.472+0000 7f78b1a81640 10 monclient(hunting): authenticate will time out at 1182.557403s 2024-03-29T11:54:35.481 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.472+0000 7f78aa7fc640 1 -- 172.21.15.98:0/2149168557 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2092009292 0 0) 0x7f78ac145eb0 con 0x7f78ac13f650 2024-03-29T11:54:35.481 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.472+0000 7f78aa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:54:35.481 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.472+0000 7f78aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:54:35.481 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.472+0000 7f78aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:54:35.481 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.472+0000 7f78aa7fc640 10 monclient(hunting): my global_id is 4472 2024-03-29T11:54:35.481 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.472+0000 7f78aa7fc640 1 -- 172.21.15.98:0/2149168557 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7894003520 con 0x7f78ac13f650 2024-03-29T11:54:35.481 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.472+0000 7f78aa7fc640 1 -- 172.21.15.98:0/2149168557 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3437441625 0 0) 0x7f78ac145bd0 con 0x7f78ac1457f0 2024-03-29T11:54:35.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.472+0000 7f78aa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:54:35.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.472+0000 7f78aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:54:35.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.472+0000 7f78aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:54:35.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.472+0000 7f78aa7fc640 10 monclient(hunting): my global_id is 4462 2024-03-29T11:54:35.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.472+0000 7f78aa7fc640 1 -- 172.21.15.98:0/2149168557 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78ac145eb0 con 0x7f78ac1457f0 2024-03-29T11:54:35.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.472+0000 7f78aa7fc640 1 -- 172.21.15.98:0/2149168557 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2456618557 0 0) 0x7f78ac0aaef0 con 0x7f78ac0a9d70 2024-03-29T11:54:35.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.472+0000 7f78aa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:54:35.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.472+0000 7f78aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:54:35.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.472+0000 7f78aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:54:35.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.472+0000 7f78aa7fc640 10 monclient(hunting): my global_id is 4437 2024-03-29T11:54:35.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.472+0000 7f78aa7fc640 1 -- 172.21.15.98:0/2149168557 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78ac145bd0 con 0x7f78ac0a9d70 2024-03-29T11:54:35.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.472+0000 7f78aa7fc640 1 -- 172.21.15.98:0/2149168557 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1953279943 0 0) 0x7f7894003520 con 0x7f78ac13f650 2024-03-29T11:54:35.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.472+0000 7f78aa7fc640 1 -- 172.21.15.98:0/2149168557 >> v1:172.21.15.121:6789/0 conn(0x7f78ac1457f0 legacy=0x7f78ac13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:35.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.472+0000 7f78aa7fc640 1 -- 172.21.15.98:0/2149168557 >> v1:172.21.15.98:6789/0 conn(0x7f78ac0a9d70 legacy=0x7f78ac14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:35.483 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.472+0000 7f78aa7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:54:35.483 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.472+0000 7f78aa7fc640 1 monclient: found mon.noname-b 2024-03-29T11:54:35.483 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.472+0000 7f78aa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:54:35.483 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.472+0000 7f78aa7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:54:35.483 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.472+0000 7f78aa7fc640 1 -- 172.21.15.98:0/2149168557 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78ac146340 con 0x7f78ac13f650 2024-03-29T11:54:35.483 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.472+0000 7f78aa7fc640 10 monclient: _finish_auth 0 2024-03-29T11:54:35.483 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.472+0000 7f78aa7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:54:35.483 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.472+0000 7f78aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:54:35.483 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.472+0000 7f78aa7fc640 1 -- 172.21.15.98:0/2149168557 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f789c001690 con 0x7f78ac13f650 2024-03-29T11:54:35.483 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.473+0000 7f78aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:54:35.483 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.473+0000 7f78aa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:54:35.483 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.473+0000 7f78aa7fc640 10 monclient: dump: 2024-03-29T11:54:35.483 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:54:35.483 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:35.483 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:35.484 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:35.484 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:54:35.484 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:35.484 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:54:35.484 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:54:35.484 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:54:35.484 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:35.484 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.473+0000 7f78b1a81640 5 monclient: authenticate success, global_id 4472 2024-03-29T11:54:35.484 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.473+0000 7f78b1a81640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:54:35.492 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.474+0000 7f78aa7fc640 1 -- 172.21.15.98:0/2149168557 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f789c0017f0 con 0x7f78ac13f650 2024-03-29T11:54:35.493 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.474+0000 7f78aa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:54:35.493 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.474+0000 7f78aa7fc640 1 -- 172.21.15.98:0/2149168557 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f789c001c00 con 0x7f78ac13f650 2024-03-29T11:54:35.493 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.474+0000 7f78aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:54:35.493 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.474+0000 7f78aa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:54:35.493 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.474+0000 7f78aa7fc640 10 monclient: dump: 2024-03-29T11:54:35.493 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:54:35.493 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:35.493 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:35.493 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:35.493 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:54:35.493 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:35.493 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:54:35.494 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:54:35.494 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:54:35.494 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:35.494 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.474+0000 7f78b1a81640 10 monclient: get_monmap_and_config success 2024-03-29T11:54:35.494 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.474+0000 7f78b1a81640 4 set_mon_vals no callback set 2024-03-29T11:54:35.494 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.475+0000 7f78b1a81640 10 monclient: shutdown 2024-03-29T11:54:35.494 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.475+0000 7f78b1a81640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:54:35.494 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.475+0000 7f78b1a81640 1 -- 172.21.15.98:0/2149168557 >> v1:172.21.15.98:6790/0 conn(0x7f78ac13f650 legacy=0x7f78ac0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:35.494 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.475+0000 7f78b1a81640 1 -- 172.21.15.98:0/2149168557 shutdown_connections 2024-03-29T11:54:35.494 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.475+0000 7f78b1a81640 1 -- 172.21.15.98:0/2149168557 >> 172.21.15.98:0/2149168557 conn(0x7f78ac066a00 msgr2=0x7f78ac066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:54:35.494 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.475+0000 7f78b1a81640 1 -- 172.21.15.98:0/2149168557 shutdown_connections 2024-03-29T11:54:35.494 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.475+0000 7f78b1a81640 1 -- 172.21.15.98:0/2149168557 wait complete. 2024-03-29T11:54:35.494 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.476+0000 7f78b1a81640 10 monclient: build_initial_monmap 2024-03-29T11:54:35.494 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.476+0000 7f78b1a81640 10 monclient: monmap: 2024-03-29T11:54:35.494 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:54:35.495 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:54:35.495 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:54:35.495 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:54:35.495 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:54:35.495 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:35.495 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:54:35.495 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:54:35.495 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:54:35.495 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:35.495 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.476+0000 7f78b1a81640 1 librados: starting msgr at 2024-03-29T11:54:35.495 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.476+0000 7f78b1a81640 1 librados: starting objecter 2024-03-29T11:54:35.495 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.476+0000 7f78b1a81640 1 Processor -- start 2024-03-29T11:54:35.495 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.477+0000 7f78b1a81640 1 -- start start 2024-03-29T11:54:35.495 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.477+0000 7f78b1a81640 1 librados: setting wanted keys 2024-03-29T11:54:35.496 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.477+0000 7f78b1a81640 1 librados: calling monclient init 2024-03-29T11:54:35.496 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.477+0000 7f78b1a81640 10 monclient: init 2024-03-29T11:54:35.496 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.479+0000 7f78b1a81640 10 monclient: _reopen_session rank -1 2024-03-29T11:54:35.496 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.479+0000 7f78b1a81640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:54:35.496 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.479+0000 7f78b1a81640 10 monclient(hunting): picked mon.noname-b con 0x7f78ac0a9d70 addr v1:172.21.15.121:6789/0 2024-03-29T11:54:35.496 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.479+0000 7f78b1a81640 10 monclient(hunting): picked mon.noname-a con 0x7f78ac13f650 addr v1:172.21.15.98:6789/0 2024-03-29T11:54:35.496 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.479+0000 7f78b1a81640 10 monclient(hunting): picked mon.noname-c con 0x7f78ac1457f0 addr v1:172.21.15.98:6790/0 2024-03-29T11:54:35.496 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.479+0000 7f78b1a81640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78ac06ee30 con 0x7f78ac13f650 2024-03-29T11:54:35.496 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.479+0000 7f78b1a81640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78ac06dfc0 con 0x7f78ac0a9d70 2024-03-29T11:54:35.496 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.479+0000 7f78aaffd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f78ac13f650 0x7f78ac136dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:59366/0 (socket says 172.21.15.98:59366) 2024-03-29T11:54:35.496 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.479+0000 7f78aaffd640 1 -- 172.21.15.98:0/874812709 learned_addr learned my addr 172.21.15.98:0/874812709 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:54:35.496 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.479+0000 7f7893fff640 10 client.?.objecter ms_handle_connect 0x7f78ac13f650 2024-03-29T11:54:35.496 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.479+0000 7f7893fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:54:35.496 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.479+0000 7f78b1a81640 1 -- 172.21.15.98:0/874812709 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78ac1404b0 con 0x7f78ac1457f0 2024-03-29T11:54:35.497 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.479+0000 7f78b1a81640 10 monclient(hunting): _renew_subs 2024-03-29T11:54:35.497 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.479+0000 7f78b1a81640 10 monclient(hunting): authenticate will time out at 1182.564935s 2024-03-29T11:54:35.497 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.479+0000 7f7893fff640 10 client.?.objecter ms_handle_connect 0x7f78ac1457f0 2024-03-29T11:54:35.497 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.479+0000 7f7893fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:54:35.497 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.480+0000 7f7893fff640 10 client.?.objecter ms_handle_connect 0x7f78ac0a9d70 2024-03-29T11:54:35.497 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.480+0000 7f7893fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:54:35.497 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.480+0000 7f7893fff640 1 -- 172.21.15.98:0/874812709 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4125391311 0 0) 0x7f78ac06ee30 con 0x7f78ac13f650 2024-03-29T11:54:35.497 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.480+0000 7f7893fff640 10 monclient(hunting): _negotiate 2024-03-29T11:54:35.497 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.480+0000 7f7893fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:54:35.497 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.480+0000 7f7893fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:54:35.497 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.480+0000 7f7893fff640 10 monclient(hunting): my global_id is 4440 2024-03-29T11:54:35.497 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.480+0000 7f7893fff640 1 -- 172.21.15.98:0/874812709 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7898004eb0 con 0x7f78ac13f650 2024-03-29T11:54:35.497 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.480+0000 7f7893fff640 1 -- 172.21.15.98:0/874812709 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2000006657 0 0) 0x7f78ac1404b0 con 0x7f78ac1457f0 2024-03-29T11:54:35.497 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.480+0000 7f7893fff640 10 monclient(hunting): _negotiate 2024-03-29T11:54:35.497 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.480+0000 7f7893fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:54:35.497 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.480+0000 7f7893fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:54:35.498 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.480+0000 7f7893fff640 10 monclient(hunting): my global_id is 4475 2024-03-29T11:54:35.498 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.480+0000 7f7893fff640 1 -- 172.21.15.98:0/874812709 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78ac06ee30 con 0x7f78ac1457f0 2024-03-29T11:54:35.498 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.480+0000 7f7893fff640 1 -- 172.21.15.98:0/874812709 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2850149613 0 0) 0x7f78ac06dfc0 con 0x7f78ac0a9d70 2024-03-29T11:54:35.498 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.480+0000 7f7893fff640 10 monclient(hunting): _negotiate 2024-03-29T11:54:35.498 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.480+0000 7f7893fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:54:35.498 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.480+0000 7f7893fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:54:35.498 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.480+0000 7f7893fff640 10 monclient(hunting): my global_id is 4465 2024-03-29T11:54:35.498 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.480+0000 7f7893fff640 1 -- 172.21.15.98:0/874812709 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78ac1404b0 con 0x7f78ac0a9d70 2024-03-29T11:54:35.498 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.481+0000 7f7893fff640 1 -- 172.21.15.98:0/874812709 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2093934465 0 0) 0x7f78ac06ee30 con 0x7f78ac1457f0 2024-03-29T11:54:35.498 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.481+0000 7f7893fff640 1 -- 172.21.15.98:0/874812709 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78ac06dfc0 con 0x7f78ac1457f0 2024-03-29T11:54:35.498 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.481+0000 7f7893fff640 1 -- 172.21.15.98:0/874812709 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1946968293 0 0) 0x7f78ac1404b0 con 0x7f78ac0a9d70 2024-03-29T11:54:35.498 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.481+0000 7f7893fff640 1 -- 172.21.15.98:0/874812709 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78ac06ee30 con 0x7f78ac0a9d70 2024-03-29T11:54:35.498 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.481+0000 7f7893fff640 1 -- 172.21.15.98:0/874812709 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f78a4001c30 con 0x7f78ac1457f0 2024-03-29T11:54:35.499 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.481+0000 7f7893fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:54:35.499 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.481+0000 7f7893fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:54:35.499 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.481+0000 7f7893fff640 10 monclient(hunting): dump: 2024-03-29T11:54:35.499 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:54:35.499 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:35.499 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:35.499 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:35.499 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:54:35.499 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:35.499 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:54:35.499 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:54:35.499 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:54:35.499 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:35.499 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.481+0000 7f7893fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:54:35.499 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.481+0000 7f7893fff640 1 -- 172.21.15.98:0/874812709 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f78a00017d0 con 0x7f78ac0a9d70 2024-03-29T11:54:35.500 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.481+0000 7f7893fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:54:35.500 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.481+0000 7f7893fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:54:35.500 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.481+0000 7f7893fff640 10 monclient(hunting): dump: 2024-03-29T11:54:35.500 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:54:35.500 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:35.500 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:35.500 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:35.500 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:54:35.500 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:35.500 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:54:35.500 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:54:35.500 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:54:35.500 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:35.500 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.481+0000 7f7893fff640 1 -- 172.21.15.98:0/874812709 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2226420842 0 0) 0x7f7898004eb0 con 0x7f78ac13f650 2024-03-29T11:54:35.500 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.481+0000 7f7893fff640 1 -- 172.21.15.98:0/874812709 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78ac1404b0 con 0x7f78ac13f650 2024-03-29T11:54:35.501 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.481+0000 7f7893fff640 1 -- 172.21.15.98:0/874812709 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f789c001bc0 con 0x7f78ac13f650 2024-03-29T11:54:35.501 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.481+0000 7f7893fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:54:35.501 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.481+0000 7f7893fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:54:35.501 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.481+0000 7f7893fff640 10 monclient(hunting): dump: 2024-03-29T11:54:35.501 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:54:35.501 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:35.501 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:35.501 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:35.501 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:54:35.501 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:35.501 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:54:35.501 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:54:35.501 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:54:35.501 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:35.501 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.481+0000 7f7893fff640 1 -- 172.21.15.98:0/874812709 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2689120063 0 0) 0x7f78ac06dfc0 con 0x7f78ac1457f0 2024-03-29T11:54:35.502 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.481+0000 7f7893fff640 1 -- 172.21.15.98:0/874812709 >> v1:172.21.15.121:6789/0 conn(0x7f78ac0a9d70 legacy=0x7f78ac13fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:35.502 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.481+0000 7f7893fff640 1 -- 172.21.15.98:0/874812709 >> v1:172.21.15.98:6789/0 conn(0x7f78ac13f650 legacy=0x7f78ac136dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:35.502 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.481+0000 7f7893fff640 10 monclient: _finish_hunting 0 2024-03-29T11:54:35.502 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.481+0000 7f7893fff640 1 monclient: found mon.c 2024-03-29T11:54:35.502 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.481+0000 7f7893fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:54:35.502 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.481+0000 7f7893fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:54:35.502 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.481+0000 7f7893fff640 1 -- 172.21.15.98:0/874812709 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78ac06f080 con 0x7f78ac1457f0 2024-03-29T11:54:35.502 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.481+0000 7f7893fff640 10 monclient: _finish_auth 0 2024-03-29T11:54:35.502 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.481+0000 7f7893fff640 10 monclient: _check_auth_tickets 2024-03-29T11:54:35.502 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.481+0000 7f7893fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:54:35.502 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.481+0000 7f78b1a81640 5 monclient: authenticate success, global_id 4475 2024-03-29T11:54:35.502 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.481+0000 7f78b1a81640 10 monclient: _renew_subs 2024-03-29T11:54:35.502 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.481+0000 7f78b1a81640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:54:35.502 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.481+0000 7f78b1a81640 1 -- 172.21.15.98:0/874812709 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78ac06f360 con 0x7f78ac1457f0 2024-03-29T11:54:35.503 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.481+0000 7f78b1a81640 10 client.4475.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:54:35.503 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.481+0000 7f78b1a81640 10 monclient: _renew_subs 2024-03-29T11:54:35.503 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.481+0000 7f78b1a81640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:54:35.503 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.481+0000 7f78b1a81640 1 -- 172.21.15.98:0/874812709 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f78ac0b1660 con 0x7f78ac1457f0 2024-03-29T11:54:35.503 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.481+0000 7f78b1a81640 1 librados: init done 2024-03-29T11:54:35.503 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.482+0000 7f7893fff640 1 -- 172.21.15.98:0/874812709 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f78a4003970 con 0x7f78ac1457f0 2024-03-29T11:54:35.503 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.482+0000 7f7893fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:54:35.503 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.482+0000 7f7893fff640 1 -- 172.21.15.98:0/874812709 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f78a4003c80 con 0x7f78ac1457f0 2024-03-29T11:54:35.503 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.482+0000 7f7893fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:54:35.503 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.482+0000 7f7893fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:54:35.503 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.482+0000 7f7893fff640 10 monclient: dump: 2024-03-29T11:54:35.503 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:54:35.503 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:35.503 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:35.503 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:35.504 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:54:35.504 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:35.504 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:54:35.504 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:54:35.504 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:54:35.504 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:35.504 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.482+0000 7f78a9ffb640 4 set_mon_vals no callback set 2024-03-29T11:54:35.504 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.483+0000 7f78b1a81640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:54:35.504 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.483+0000 7f78b1a81640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:54:35.504 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.483+0000 7f78b1a81640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:54:35.504 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.483+0000 7f78b1a81640 1 -- 172.21.15.98:0/874812709 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f788c005080 con 0x7f78ac1457f0 2024-03-29T11:54:35.505 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.484+0000 7f7893fff640 1 -- 172.21.15.98:0/874812709 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f78a4019c50 con 0x7f78ac1457f0 2024-03-29T11:54:35.505 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.485+0000 7f7893fff640 1 -- 172.21.15.98:0/874812709 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f78a4083b20 con 0x7f78ac1457f0 2024-03-29T11:54:35.505 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.485+0000 7f7893fff640 10 client.4475.objecter ms_dispatch 0x7f78ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:54:35.505 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.485+0000 7f7893fff640 3 client.4475.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:54:35.505 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.485+0000 7f7893fff640 3 client.4475.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:54:35.505 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.485+0000 7f7893fff640 20 client.4475.objecter dump_active .. 0 homeless 2024-03-29T11:54:35.505 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.485+0000 7f78ab7fe640 10 monclient: get_auth_request con 0x7f789806e100 auth_method 0 2024-03-29T11:54:35.505 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.486+0000 7f7893fff640 10 client.4475.objecter ms_handle_connect 0x7f789806e100 2024-03-29T11:54:35.505 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.493+0000 7f7893fff640 1 -- 172.21.15.98:0/874812709 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f78a404f400 con 0x7f78ac1457f0 2024-03-29T11:54:35.506 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.493+0000 7f7893fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:54:35.506 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.493+0000 7f7893fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:54:35.856 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.848+0000 7f78b1a81640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:54:35.856 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.848+0000 7f78b1a81640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:54:35.856 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.848+0000 7f78b1a81640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:54:35.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.848+0000 7f78b1a81640 1 -- 172.21.15.98:0/874812709 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f788c005330 con 0x7f78ac1457f0 2024-03-29T11:54:35.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.849+0000 7f7893fff640 1 -- 172.21.15.98:0/874812709 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f78a4051330 con 0x7f78ac1457f0 2024-03-29T11:54:35.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.849+0000 7f7893fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:54:35.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.849+0000 7f7893fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:54:35.858 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:54:35.858 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:54:35.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.853+0000 7f78b1a81640 10 librados: watch_flush enter 2024-03-29T11:54:35.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.853+0000 7f78b1a81640 10 librados: watch_flush exit 2024-03-29T11:54:35.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.853+0000 7f78b1a81640 20 client.4475.objecter shutdown clearing up homeless session... 2024-03-29T11:54:35.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.853+0000 7f78b1a81640 10 client.4475.objecter successfully canceled tick 2024-03-29T11:54:35.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.853+0000 7f78b1a81640 1 -- 172.21.15.98:0/874812709 >> v1:172.21.15.98:6816/4065186974 conn(0x7f789806e100 legacy=0x7f78980705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:35.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.853+0000 7f78b1a81640 10 monclient: shutdown 2024-03-29T11:54:35.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.853+0000 7f78b1a81640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:54:35.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.853+0000 7f78b1a81640 1 -- 172.21.15.98:0/874812709 >> v1:172.21.15.98:6790/0 conn(0x7f78ac1457f0 legacy=0x7f78ac1ef820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:35.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.853+0000 7f78b1a81640 1 -- 172.21.15.98:0/874812709 shutdown_connections 2024-03-29T11:54:35.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.854+0000 7f78b1a81640 1 -- 172.21.15.98:0/874812709 >> 172.21.15.98:0/874812709 conn(0x7f78ac066a00 msgr2=0x7f78ac0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:54:35.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.854+0000 7f78b1a81640 1 -- 172.21.15.98:0/874812709 shutdown_connections 2024-03-29T11:54:35.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.854+0000 7f78b1a81640 1 -- 172.21.15.98:0/874812709 wait complete. 2024-03-29T11:54:35.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:35.854+0000 7f78b1a81640 1 librados: shutdown 2024-03-29T11:54:35.878 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:54:42.686 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:54:42.835 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.827+0000 7f43d2364640 10 monclient: get_monmap_and_config 2024-03-29T11:54:42.836 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.827+0000 7f43d2364640 10 monclient: build_initial_monmap 2024-03-29T11:54:42.836 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.827+0000 7f43d2364640 10 monclient: monmap: 2024-03-29T11:54:42.836 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:54:42.836 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:42.836 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:54:42.828705+0000 2024-03-29T11:54:42.836 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:54:42.828705+0000 2024-03-29T11:54:42.836 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:54:42.836 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:42.836 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:54:42.836 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:54:42.836 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:54:42.837 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:42.837 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.828+0000 7f43d2364640 1 Processor -- start 2024-03-29T11:54:42.837 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.829+0000 7f43d2364640 1 -- start start 2024-03-29T11:54:42.837 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.829+0000 7f43d2364640 10 monclient: init 2024-03-29T11:54:42.838 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.829+0000 7f43d2364640 10 monclient: _reopen_session rank -1 2024-03-29T11:54:42.838 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.829+0000 7f43d2364640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:54:42.838 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.829+0000 7f43d2364640 10 monclient(hunting): picked mon.noname-c con 0x7f43cc0b0860 addr v1:172.21.15.121:6789/0 2024-03-29T11:54:42.838 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.829+0000 7f43d2364640 10 monclient(hunting): picked mon.noname-a con 0x7f43cc072450 addr v1:172.21.15.98:6789/0 2024-03-29T11:54:42.838 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.830+0000 7f43d2364640 10 monclient(hunting): picked mon.noname-b con 0x7f43cc13ee40 addr v1:172.21.15.98:6790/0 2024-03-29T11:54:42.838 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.830+0000 7f43d2364640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43cc0ad840 con 0x7f43cc072450 2024-03-29T11:54:42.838 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.830+0000 7f43d2364640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43cc13c830 con 0x7f43cc0b0860 2024-03-29T11:54:42.838 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.830+0000 7f43d2364640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43cc13c9c0 con 0x7f43cc13ee40 2024-03-29T11:54:42.838 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.830+0000 7f43d2364640 10 monclient(hunting): _renew_subs 2024-03-29T11:54:42.839 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.830+0000 7f43d2364640 10 monclient(hunting): authenticate will time out at 1189.915116s 2024-03-29T11:54:42.839 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.830+0000 7f43d0901640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f43cc13ee40 0x7f43cc13f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:34788/0 (socket says 172.21.15.98:34788) 2024-03-29T11:54:42.839 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.830+0000 7f43d0901640 1 -- 172.21.15.98:0/358291723 learned_addr learned my addr 172.21.15.98:0/358291723 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:54:42.839 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.830+0000 7f43cb7fe640 1 --1- 172.21.15.98:0/358291723 >> v1:172.21.15.98:6789/0 conn(0x7f43cc072450 0x7f43cc13c120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:58630/0 (socket says 172.21.15.98:58630) 2024-03-29T11:54:42.840 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.830+0000 7f43caffd640 1 -- 172.21.15.98:0/358291723 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3134950807 0 0) 0x7f43cc0ad840 con 0x7f43cc072450 2024-03-29T11:54:42.840 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.830+0000 7f43caffd640 10 monclient(hunting): _negotiate 2024-03-29T11:54:42.840 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.830+0000 7f43caffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:54:42.840 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.830+0000 7f43caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:54:42.840 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.830+0000 7f43caffd640 10 monclient(hunting): my global_id is 4443 2024-03-29T11:54:42.840 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.830+0000 7f43caffd640 1 -- 172.21.15.98:0/358291723 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43b4003520 con 0x7f43cc072450 2024-03-29T11:54:42.840 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.831+0000 7f43caffd640 1 -- 172.21.15.98:0/358291723 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1366455366 0 0) 0x7f43cc13c9c0 con 0x7f43cc13ee40 2024-03-29T11:54:42.840 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.831+0000 7f43caffd640 10 monclient(hunting): _negotiate 2024-03-29T11:54:42.840 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.831+0000 7f43caffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:54:42.840 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.831+0000 7f43caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:54:42.840 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.831+0000 7f43caffd640 10 monclient(hunting): my global_id is 4478 2024-03-29T11:54:42.841 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.831+0000 7f43caffd640 1 -- 172.21.15.98:0/358291723 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43cc0ad840 con 0x7f43cc13ee40 2024-03-29T11:54:42.841 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.831+0000 7f43caffd640 1 -- 172.21.15.98:0/358291723 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3158963687 0 0) 0x7f43b4003520 con 0x7f43cc072450 2024-03-29T11:54:42.841 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.831+0000 7f43caffd640 1 -- 172.21.15.98:0/358291723 >> v1:172.21.15.98:6790/0 conn(0x7f43cc13ee40 legacy=0x7f43cc13f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:42.841 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.831+0000 7f43caffd640 1 -- 172.21.15.98:0/358291723 >> v1:172.21.15.121:6789/0 conn(0x7f43cc0b0860 legacy=0x7f43cc0acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:54:42.841 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.831+0000 7f43caffd640 10 monclient: _finish_hunting 0 2024-03-29T11:54:42.841 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.831+0000 7f43caffd640 1 monclient: found mon.noname-a 2024-03-29T11:54:42.841 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.831+0000 7f43caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:54:42.841 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.831+0000 7f43caffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T11:54:42.841 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.831+0000 7f43caffd640 1 -- 172.21.15.98:0/358291723 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43cc13f970 con 0x7f43cc072450 2024-03-29T11:54:42.841 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.831+0000 7f43caffd640 10 monclient: _finish_auth 0 2024-03-29T11:54:42.841 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.831+0000 7f43caffd640 10 monclient: _check_auth_tickets 2024-03-29T11:54:42.841 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.831+0000 7f43caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:54:42.841 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.831+0000 7f43caffd640 1 -- 172.21.15.98:0/358291723 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f43c0001690 con 0x7f43cc072450 2024-03-29T11:54:42.841 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.831+0000 7f43caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:54:42.842 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.831+0000 7f43caffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:54:42.842 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.831+0000 7f43caffd640 10 monclient: dump: 2024-03-29T11:54:42.842 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:54:42.842 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:42.842 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:42.842 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:42.842 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:54:42.842 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:42.842 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:54:42.842 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:54:42.842 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:54:42.842 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:42.842 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.831+0000 7f43d2364640 5 monclient: authenticate success, global_id 4443 2024-03-29T11:54:42.842 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.831+0000 7f43d2364640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:54:42.843 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.831+0000 7f43caffd640 1 -- 172.21.15.98:0/358291723 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43c00017f0 con 0x7f43cc072450 2024-03-29T11:54:42.843 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.831+0000 7f43caffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:54:42.843 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.831+0000 7f43d2364640 10 monclient: get_monmap_and_config success 2024-03-29T11:54:42.843 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.831+0000 7f43d2364640 4 set_mon_vals no callback set 2024-03-29T11:54:42.847 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.832+0000 7f43d2364640 10 monclient: shutdown 2024-03-29T11:54:42.847 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.832+0000 7f43caffd640 1 -- 172.21.15.98:0/358291723 <== mon.0 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f43c0001c00 con 0x7f43cc072450 2024-03-29T11:54:42.847 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.832+0000 7f43caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:54:42.847 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.832+0000 7f43caffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:54:42.847 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.832+0000 7f43caffd640 10 monclient: dump: 2024-03-29T11:54:42.848 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:54:42.848 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:42.848 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:42.848 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:42.848 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:54:42.848 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:42.848 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:54:42.848 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:54:42.848 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:54:42.848 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:42.848 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.832+0000 7f43d2364640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:54:42.848 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.832+0000 7f43d2364640 1 -- 172.21.15.98:0/358291723 >> v1:172.21.15.98:6789/0 conn(0x7f43cc072450 legacy=0x7f43cc13c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:42.848 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.833+0000 7f43d2364640 1 -- 172.21.15.98:0/358291723 shutdown_connections 2024-03-29T11:54:42.848 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.833+0000 7f43d2364640 1 -- 172.21.15.98:0/358291723 >> 172.21.15.98:0/358291723 conn(0x7f43cc06b660 msgr2=0x7f43cc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:54:42.849 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.833+0000 7f43d2364640 1 -- 172.21.15.98:0/358291723 shutdown_connections 2024-03-29T11:54:42.849 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.833+0000 7f43d2364640 1 -- 172.21.15.98:0/358291723 wait complete. 2024-03-29T11:54:42.849 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.834+0000 7f43d2364640 10 monclient: build_initial_monmap 2024-03-29T11:54:42.849 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.834+0000 7f43d2364640 10 monclient: monmap: 2024-03-29T11:54:42.849 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:54:42.849 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:54:42.849 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:54:42.849 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:54:42.849 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:54:42.849 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:42.849 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:54:42.849 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:54:42.849 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:54:42.849 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:42.849 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.834+0000 7f43d2364640 1 librados: starting msgr at 2024-03-29T11:54:42.850 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.835+0000 7f43d2364640 1 librados: starting objecter 2024-03-29T11:54:42.850 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.835+0000 7f43d2364640 1 Processor -- start 2024-03-29T11:54:42.850 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.835+0000 7f43d2364640 1 -- start start 2024-03-29T11:54:42.850 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.835+0000 7f43d2364640 1 librados: setting wanted keys 2024-03-29T11:54:42.850 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.835+0000 7f43d2364640 1 librados: calling monclient init 2024-03-29T11:54:42.850 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.835+0000 7f43d2364640 10 monclient: init 2024-03-29T11:54:42.850 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.836+0000 7f43d2364640 10 monclient: _reopen_session rank -1 2024-03-29T11:54:42.850 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.836+0000 7f43d2364640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:54:42.850 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.836+0000 7f43d2364640 10 monclient(hunting): picked mon.noname-a con 0x7f43cc072450 addr v1:172.21.15.98:6789/0 2024-03-29T11:54:42.850 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.836+0000 7f43d2364640 10 monclient(hunting): picked mon.noname-c con 0x7f43cc0b0860 addr v1:172.21.15.98:6790/0 2024-03-29T11:54:42.850 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.836+0000 7f43d2364640 10 monclient(hunting): picked mon.noname-b con 0x7f43cc13ee40 addr v1:172.21.15.121:6789/0 2024-03-29T11:54:42.850 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.836+0000 7f43d2364640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43cc1db7c0 con 0x7f43cc072450 2024-03-29T11:54:42.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.836+0000 7f43d2364640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43cc1dc360 con 0x7f43cc13ee40 2024-03-29T11:54:42.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.836+0000 7f43d2364640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43cc142f40 con 0x7f43cc0b0860 2024-03-29T11:54:42.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.836+0000 7f43d2364640 10 monclient(hunting): _renew_subs 2024-03-29T11:54:42.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.836+0000 7f43d2364640 10 monclient(hunting): authenticate will time out at 1189.921635s 2024-03-29T11:54:42.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.836+0000 7f43cb7fe640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f43cc0b0860 0x7f43cc1dbc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:35076/0 (socket says 172.21.15.98:35076) 2024-03-29T11:54:42.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.836+0000 7f43cb7fe640 1 -- 172.21.15.98:0/3337871118 learned_addr learned my addr 172.21.15.98:0/3337871118 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:54:42.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.837+0000 7f43c8ff9640 10 client.?.objecter ms_handle_connect 0x7f43cc13ee40 2024-03-29T11:54:42.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.837+0000 7f43c8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:54:42.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.837+0000 7f43c8ff9640 10 client.?.objecter ms_handle_connect 0x7f43cc0b0860 2024-03-29T11:54:42.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.837+0000 7f43c8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:54:42.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.837+0000 7f43c8ff9640 10 client.?.objecter ms_handle_connect 0x7f43cc072450 2024-03-29T11:54:42.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.837+0000 7f43c8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:54:42.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.837+0000 7f43c8ff9640 1 -- 172.21.15.98:0/3337871118 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3227656073 0 0) 0x7f43cc142f40 con 0x7f43cc0b0860 2024-03-29T11:54:42.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.837+0000 7f43c8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:54:42.852 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.837+0000 7f43c8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:54:42.852 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.837+0000 7f43c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:54:42.852 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.837+0000 7f43c8ff9640 10 monclient(hunting): my global_id is 4481 2024-03-29T11:54:42.852 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.837+0000 7f43c8ff9640 1 -- 172.21.15.98:0/3337871118 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43b8004eb0 con 0x7f43cc0b0860 2024-03-29T11:54:42.852 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.837+0000 7f43c8ff9640 1 -- 172.21.15.98:0/3337871118 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2079086107 0 0) 0x7f43cc1dc360 con 0x7f43cc13ee40 2024-03-29T11:54:42.852 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.837+0000 7f43c8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:54:42.852 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.837+0000 7f43c8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:54:42.852 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.837+0000 7f43c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:54:42.852 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.837+0000 7f43c8ff9640 10 monclient(hunting): my global_id is 4468 2024-03-29T11:54:42.852 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.837+0000 7f43c8ff9640 1 -- 172.21.15.98:0/3337871118 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43cc142f40 con 0x7f43cc13ee40 2024-03-29T11:54:42.852 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.837+0000 7f43c8ff9640 1 -- 172.21.15.98:0/3337871118 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2462205593 0 0) 0x7f43cc1db7c0 con 0x7f43cc072450 2024-03-29T11:54:42.852 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.837+0000 7f43c8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:54:42.852 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.837+0000 7f43c8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:54:42.852 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.837+0000 7f43c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:54:42.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.837+0000 7f43c8ff9640 10 monclient(hunting): my global_id is 4446 2024-03-29T11:54:42.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.837+0000 7f43c8ff9640 1 -- 172.21.15.98:0/3337871118 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43cc1dc360 con 0x7f43cc072450 2024-03-29T11:54:42.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43c8ff9640 1 -- 172.21.15.98:0/3337871118 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1848073076 0 0) 0x7f43b8004eb0 con 0x7f43cc0b0860 2024-03-29T11:54:42.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43c8ff9640 1 -- 172.21.15.98:0/3337871118 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43cc1db7c0 con 0x7f43cc0b0860 2024-03-29T11:54:42.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43c8ff9640 1 -- 172.21.15.98:0/3337871118 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2538554546 0 0) 0x7f43cc1dc360 con 0x7f43cc072450 2024-03-29T11:54:42.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43c8ff9640 1 -- 172.21.15.98:0/3337871118 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43b8004eb0 con 0x7f43cc072450 2024-03-29T11:54:42.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43c8ff9640 1 -- 172.21.15.98:0/3337871118 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f43c0001bc0 con 0x7f43cc0b0860 2024-03-29T11:54:42.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43c8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:54:42.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43c8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:54:42.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43c8ff9640 10 monclient(hunting): dump: 2024-03-29T11:54:42.853 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:54:42.853 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:42.853 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:42.853 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:42.853 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:54:42.854 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:42.854 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:54:42.854 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:54:42.854 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:54:42.854 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:42.854 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43c8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:54:42.854 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43c8ff9640 1 -- 172.21.15.98:0/3337871118 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3338931849 0 0) 0x7f43cc142f40 con 0x7f43cc13ee40 2024-03-29T11:54:42.854 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43c8ff9640 1 -- 172.21.15.98:0/3337871118 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43cc1dc360 con 0x7f43cc13ee40 2024-03-29T11:54:42.854 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43c8ff9640 1 -- 172.21.15.98:0/3337871118 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f43c4001810 con 0x7f43cc072450 2024-03-29T11:54:42.854 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43c8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:54:42.854 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43c8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:54:42.854 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43c8ff9640 10 monclient(hunting): dump: 2024-03-29T11:54:42.854 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:54:42.854 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:42.855 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:42.855 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:42.855 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:54:42.855 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:42.855 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:54:42.855 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:54:42.855 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:54:42.855 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:42.855 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43c8ff9640 1 -- 172.21.15.98:0/3337871118 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f43bc001700 con 0x7f43cc13ee40 2024-03-29T11:54:42.855 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43c8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:54:42.855 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43c8ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:54:42.855 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43c8ff9640 10 monclient(hunting): dump: 2024-03-29T11:54:42.855 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:54:42.855 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:42.855 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:42.856 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:42.856 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:54:42.856 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:42.856 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:54:42.856 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:54:42.856 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:54:42.856 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:42.856 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43c8ff9640 1 -- 172.21.15.98:0/3337871118 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3421731702 0 0) 0x7f43b8004eb0 con 0x7f43cc072450 2024-03-29T11:54:42.856 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43c8ff9640 1 -- 172.21.15.98:0/3337871118 >> v1:172.21.15.98:6790/0 conn(0x7f43cc0b0860 legacy=0x7f43cc1dbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:42.856 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43c8ff9640 1 -- 172.21.15.98:0/3337871118 >> v1:172.21.15.121:6789/0 conn(0x7f43cc13ee40 legacy=0x7f43cc1da7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:42.856 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43c8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:54:42.856 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43c8ff9640 1 monclient: found mon.a 2024-03-29T11:54:42.856 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43c8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:54:42.856 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43c8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:54:42.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43c8ff9640 1 -- 172.21.15.98:0/3337871118 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43cc140bc0 con 0x7f43cc072450 2024-03-29T11:54:42.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43c8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:54:42.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43c8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:54:42.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43c8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:54:42.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43d2364640 5 monclient: authenticate success, global_id 4446 2024-03-29T11:54:42.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43d2364640 10 monclient: _renew_subs 2024-03-29T11:54:42.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43d2364640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:54:42.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43d2364640 1 -- 172.21.15.98:0/3337871118 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43cc13cec0 con 0x7f43cc072450 2024-03-29T11:54:42.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43d2364640 10 client.4446.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:54:42.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43d2364640 10 monclient: _renew_subs 2024-03-29T11:54:42.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43d2364640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:54:42.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43c8ff9640 1 -- 172.21.15.98:0/3337871118 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43c4001d20 con 0x7f43cc072450 2024-03-29T11:54:42.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43d2364640 1 -- 172.21.15.98:0/3337871118 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f43cc1daef0 con 0x7f43cc072450 2024-03-29T11:54:42.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43c8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:54:42.858 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43c8ff9640 1 -- 172.21.15.98:0/3337871118 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f43c4003b00 con 0x7f43cc072450 2024-03-29T11:54:42.858 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43c8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:54:42.858 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43c8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:54:42.858 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43c8ff9640 10 monclient: dump: 2024-03-29T11:54:42.858 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:54:42.858 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:42.858 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:42.858 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:42.858 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:54:42.858 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:42.858 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:54:42.858 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:54:42.858 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:54:42.858 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:42.858 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43d2364640 1 librados: init done 2024-03-29T11:54:42.859 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.838+0000 7f43c9ffb640 4 set_mon_vals no callback set 2024-03-29T11:54:42.860 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.840+0000 7f43d2364640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:54:42.860 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.840+0000 7f43d2364640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:54:42.860 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.840+0000 7f43d2364640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:54:42.860 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.840+0000 7f43d2364640 1 -- 172.21.15.98:0/3337871118 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43cc140e10 con 0x7f43cc072450 2024-03-29T11:54:42.860 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.841+0000 7f43c8ff9640 1 -- 172.21.15.98:0/3337871118 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f43c4019ad0 con 0x7f43cc072450 2024-03-29T11:54:42.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.841+0000 7f43c8ff9640 1 -- 172.21.15.98:0/3337871118 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f43c4083980 con 0x7f43cc072450 2024-03-29T11:54:42.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.841+0000 7f43c8ff9640 10 client.4446.objecter ms_dispatch 0x7f43cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:54:42.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.841+0000 7f43c8ff9640 3 client.4446.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:54:42.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.841+0000 7f43c8ff9640 3 client.4446.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:54:42.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.841+0000 7f43c8ff9640 20 client.4446.objecter dump_active .. 0 homeless 2024-03-29T11:54:42.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.842+0000 7f43cb7fe640 10 monclient: get_auth_request con 0x7f43b806e0d0 auth_method 0 2024-03-29T11:54:42.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.842+0000 7f43c8ff9640 10 client.4446.objecter ms_handle_connect 0x7f43b806e0d0 2024-03-29T11:54:42.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.848+0000 7f43c8ff9640 1 -- 172.21.15.98:0/3337871118 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f43c404f260 con 0x7f43cc072450 2024-03-29T11:54:42.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.848+0000 7f43c8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:54:42.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:42.848+0000 7f43c8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:54:43.231 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:43.223+0000 7f43d2364640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:54:43.231 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:43.223+0000 7f43d2364640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:54:43.231 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:43.223+0000 7f43d2364640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:54:43.231 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:43.223+0000 7f43d2364640 1 -- 172.21.15.98:0/3337871118 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43cc0b0c40 con 0x7f43cc072450 2024-03-29T11:54:43.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:43.223+0000 7f43c8ff9640 1 -- 172.21.15.98:0/3337871118 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f43c4051190 con 0x7f43cc072450 2024-03-29T11:54:43.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:43.223+0000 7f43c8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:54:43.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:43.223+0000 7f43c8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:54:43.232 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:54:43.232 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:54:43.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:43.227+0000 7f43d2364640 10 librados: watch_flush enter 2024-03-29T11:54:43.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:43.227+0000 7f43d2364640 10 librados: watch_flush exit 2024-03-29T11:54:43.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:43.227+0000 7f43d2364640 20 client.4446.objecter shutdown clearing up homeless session... 2024-03-29T11:54:43.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:43.227+0000 7f43d2364640 10 client.4446.objecter successfully canceled tick 2024-03-29T11:54:43.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:43.227+0000 7f43d2364640 1 -- 172.21.15.98:0/3337871118 >> v1:172.21.15.98:6816/4065186974 conn(0x7f43b806e0d0 legacy=0x7f43b8070590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:43.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:43.227+0000 7f43d2364640 10 monclient: shutdown 2024-03-29T11:54:43.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:43.227+0000 7f43d2364640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:54:43.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:43.227+0000 7f43d2364640 1 -- 172.21.15.98:0/3337871118 >> v1:172.21.15.98:6789/0 conn(0x7f43cc072450 legacy=0x7f43cc1418b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:43.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:43.227+0000 7f43d2364640 1 -- 172.21.15.98:0/3337871118 shutdown_connections 2024-03-29T11:54:43.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:43.227+0000 7f43d2364640 1 -- 172.21.15.98:0/3337871118 >> 172.21.15.98:0/3337871118 conn(0x7f43cc06b660 msgr2=0x7f43cc0a9d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:54:43.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:43.228+0000 7f43d2364640 1 -- 172.21.15.98:0/3337871118 shutdown_connections 2024-03-29T11:54:43.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:43.228+0000 7f43d2364640 1 -- 172.21.15.98:0/3337871118 wait complete. 2024-03-29T11:54:43.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:43.228+0000 7f43d2364640 1 librados: shutdown 2024-03-29T11:54:43.250 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:54:50.057 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:54:50.223 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.215+0000 7f0b938ae640 10 monclient: get_monmap_and_config 2024-03-29T11:54:50.224 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.215+0000 7f0b938ae640 10 monclient: build_initial_monmap 2024-03-29T11:54:50.224 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.215+0000 7f0b938ae640 10 monclient: monmap: 2024-03-29T11:54:50.224 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:54:50.224 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:50.224 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:54:50.216511+0000 2024-03-29T11:54:50.224 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:54:50.216511+0000 2024-03-29T11:54:50.224 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:54:50.224 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:50.224 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:54:50.224 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:54:50.224 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:54:50.224 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:50.225 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.217+0000 7f0b938ae640 1 Processor -- start 2024-03-29T11:54:50.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.217+0000 7f0b938ae640 1 -- start start 2024-03-29T11:54:50.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.217+0000 7f0b938ae640 10 monclient: init 2024-03-29T11:54:50.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.218+0000 7f0b938ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:54:50.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.218+0000 7f0b938ae640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:54:50.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.218+0000 7f0b938ae640 10 monclient(hunting): picked mon.noname-b con 0x7f0b8c0b60c0 addr v1:172.21.15.98:6790/0 2024-03-29T11:54:50.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.218+0000 7f0b938ae640 10 monclient(hunting): picked mon.noname-c con 0x7f0b8c0b64a0 addr v1:172.21.15.121:6789/0 2024-03-29T11:54:50.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.218+0000 7f0b938ae640 10 monclient(hunting): picked mon.noname-a con 0x7f0b8c070840 addr v1:172.21.15.98:6789/0 2024-03-29T11:54:50.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.218+0000 7f0b938ae640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b8c0720d0 con 0x7f0b8c070840 2024-03-29T11:54:50.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.218+0000 7f0b938ae640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b8c1439a0 con 0x7f0b8c0b64a0 2024-03-29T11:54:50.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.218+0000 7f0b938ae640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b8c143690 con 0x7f0b8c0b60c0 2024-03-29T11:54:50.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.218+0000 7f0b938ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:54:50.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.218+0000 7f0b938ae640 10 monclient(hunting): authenticate will time out at 1197.303647s 2024-03-29T11:54:50.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.224+0000 7f0b90e49640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f0b8c0b64a0 0x7f0b8c070130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:33123/0 (socket says 172.21.15.98:33123) 2024-03-29T11:54:50.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.224+0000 7f0b90e49640 1 -- 172.21.15.98:0/928432760 learned_addr learned my addr 172.21.15.98:0/928432760 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:54:50.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.224+0000 7f0b83fff640 1 -- 172.21.15.98:0/928432760 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1618565412 0 0) 0x7f0b8c1439a0 con 0x7f0b8c0b64a0 2024-03-29T11:54:50.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.224+0000 7f0b83fff640 10 monclient(hunting): _negotiate 2024-03-29T11:54:50.233 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.224+0000 7f0b83fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:54:50.233 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.224+0000 7f0b83fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:54:50.233 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.224+0000 7f0b83fff640 10 monclient(hunting): my global_id is 4471 2024-03-29T11:54:50.233 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.224+0000 7f0b83fff640 1 -- 172.21.15.98:0/928432760 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b78003520 con 0x7f0b8c0b64a0 2024-03-29T11:54:50.233 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.224+0000 7f0b83fff640 1 -- 172.21.15.98:0/928432760 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1960674112 0 0) 0x7f0b8c143690 con 0x7f0b8c0b60c0 2024-03-29T11:54:50.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.224+0000 7f0b83fff640 10 monclient(hunting): _negotiate 2024-03-29T11:54:50.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.224+0000 7f0b83fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:54:50.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.224+0000 7f0b83fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:54:50.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.224+0000 7f0b83fff640 10 monclient(hunting): my global_id is 4484 2024-03-29T11:54:50.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.225+0000 7f0b83fff640 1 -- 172.21.15.98:0/928432760 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b8c1439a0 con 0x7f0b8c0b60c0 2024-03-29T11:54:50.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.225+0000 7f0b83fff640 1 -- 172.21.15.98:0/928432760 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3477486285 0 0) 0x7f0b78003520 con 0x7f0b8c0b64a0 2024-03-29T11:54:50.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.225+0000 7f0b83fff640 1 -- 172.21.15.98:0/928432760 >> v1:172.21.15.98:6790/0 conn(0x7f0b8c0b60c0 legacy=0x7f0b8c139750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:50.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.225+0000 7f0b83fff640 1 -- 172.21.15.98:0/928432760 >> v1:172.21.15.98:6789/0 conn(0x7f0b8c070840 legacy=0x7f0b8c14b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:50.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.225+0000 7f0b83fff640 10 monclient: _finish_hunting 0 2024-03-29T11:54:50.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.225+0000 7f0b83fff640 1 monclient: found mon.noname-c 2024-03-29T11:54:50.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.225+0000 7f0b83fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:54:50.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.225+0000 7f0b83fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:54:50.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.225+0000 7f0b83fff640 1 -- 172.21.15.98:0/928432760 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b8c06dca0 con 0x7f0b8c0b64a0 2024-03-29T11:54:50.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.225+0000 7f0b83fff640 10 monclient: _finish_auth 0 2024-03-29T11:54:50.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.225+0000 7f0b83fff640 10 monclient: _check_auth_tickets 2024-03-29T11:54:50.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.225+0000 7f0b83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:54:50.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.225+0000 7f0b83fff640 1 -- 172.21.15.98:0/928432760 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f0b88001690 con 0x7f0b8c0b64a0 2024-03-29T11:54:50.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.225+0000 7f0b83fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:54:50.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.225+0000 7f0b83fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:54:50.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.225+0000 7f0b83fff640 10 monclient: dump: 2024-03-29T11:54:50.238 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:54:50.238 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:50.238 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:50.238 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:50.238 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:54:50.238 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:50.238 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:54:50.238 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:54:50.238 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:54:50.238 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:50.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.225+0000 7f0b938ae640 5 monclient: authenticate success, global_id 4471 2024-03-29T11:54:50.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.225+0000 7f0b938ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:54:50.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.225+0000 7f0b83fff640 1 -- 172.21.15.98:0/928432760 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b880017f0 con 0x7f0b8c0b64a0 2024-03-29T11:54:50.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.225+0000 7f0b83fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:54:50.239 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.225+0000 7f0b83fff640 1 -- 172.21.15.98:0/928432760 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f0b88001c00 con 0x7f0b8c0b64a0 2024-03-29T11:54:50.239 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.225+0000 7f0b83fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:54:50.239 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.225+0000 7f0b83fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:54:50.239 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.225+0000 7f0b83fff640 10 monclient: dump: 2024-03-29T11:54:50.239 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:54:50.239 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:50.239 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:50.239 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:50.239 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:54:50.239 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:50.239 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:54:50.239 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:54:50.239 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:54:50.239 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:50.239 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.227+0000 7f0b938ae640 10 monclient: get_monmap_and_config success 2024-03-29T11:54:50.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.227+0000 7f0b938ae640 4 set_mon_vals no callback set 2024-03-29T11:54:50.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.228+0000 7f0b938ae640 10 monclient: shutdown 2024-03-29T11:54:50.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.228+0000 7f0b938ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:54:50.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.228+0000 7f0b938ae640 1 -- 172.21.15.98:0/928432760 >> v1:172.21.15.121:6789/0 conn(0x7f0b8c0b64a0 legacy=0x7f0b8c070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:50.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.229+0000 7f0b938ae640 1 -- 172.21.15.98:0/928432760 shutdown_connections 2024-03-29T11:54:50.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.229+0000 7f0b938ae640 1 -- 172.21.15.98:0/928432760 >> 172.21.15.98:0/928432760 conn(0x7f0b8c067120 msgr2=0x7f0b8c0b59b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:54:50.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.232+0000 7f0b938ae640 1 -- 172.21.15.98:0/928432760 shutdown_connections 2024-03-29T11:54:50.241 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.232+0000 7f0b938ae640 1 -- 172.21.15.98:0/928432760 wait complete. 2024-03-29T11:54:50.242 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.233+0000 7f0b938ae640 10 monclient: build_initial_monmap 2024-03-29T11:54:50.242 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.233+0000 7f0b938ae640 10 monclient: monmap: 2024-03-29T11:54:50.242 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:54:50.242 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:54:50.242 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:54:50.242 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:54:50.243 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:54:50.243 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:50.243 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:54:50.243 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:54:50.243 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:54:50.243 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:50.243 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.234+0000 7f0b938ae640 1 librados: starting msgr at 2024-03-29T11:54:50.243 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.234+0000 7f0b938ae640 1 librados: starting objecter 2024-03-29T11:54:50.243 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.234+0000 7f0b938ae640 1 Processor -- start 2024-03-29T11:54:50.243 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.234+0000 7f0b938ae640 1 -- start start 2024-03-29T11:54:50.243 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.234+0000 7f0b938ae640 1 librados: setting wanted keys 2024-03-29T11:54:50.243 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.234+0000 7f0b938ae640 1 librados: calling monclient init 2024-03-29T11:54:50.243 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.234+0000 7f0b938ae640 10 monclient: init 2024-03-29T11:54:50.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.235+0000 7f0b938ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:54:50.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.235+0000 7f0b938ae640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:54:50.246 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.235+0000 7f0b938ae640 10 monclient(hunting): picked mon.noname-c con 0x7f0b8c070840 addr v1:172.21.15.98:6790/0 2024-03-29T11:54:50.249 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.239+0000 7f0b938ae640 10 monclient(hunting): picked mon.noname-b con 0x7f0b8c0b60c0 addr v1:172.21.15.121:6789/0 2024-03-29T11:54:50.249 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.241+0000 7f0b938ae640 10 monclient(hunting): picked mon.noname-a con 0x7f0b8c1dfe10 addr v1:172.21.15.98:6789/0 2024-03-29T11:54:50.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.241+0000 7f0b938ae640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b8c144040 con 0x7f0b8c1dfe10 2024-03-29T11:54:50.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.241+0000 7f0b938ae640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b8c0ab2e0 con 0x7f0b8c0b60c0 2024-03-29T11:54:50.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.241+0000 7f0b938ae640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b8c128470 con 0x7f0b8c070840 2024-03-29T11:54:50.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.241+0000 7f0b938ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:54:50.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.241+0000 7f0b938ae640 10 monclient(hunting): authenticate will time out at 1197.326500s 2024-03-29T11:54:50.254 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.241+0000 7f0b9164a640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f0b8c070840 0x7f0b8c140150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:51903/0 (socket says 172.21.15.98:51903) 2024-03-29T11:54:50.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.245+0000 7f0b9164a640 1 -- 172.21.15.98:0/2625496386 learned_addr learned my addr 172.21.15.98:0/2625496386 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:54:50.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.245+0000 7f0b81ffb640 10 client.?.objecter ms_handle_connect 0x7f0b8c070840 2024-03-29T11:54:50.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.245+0000 7f0b81ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:54:50.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.245+0000 7f0b81ffb640 1 -- 172.21.15.98:0/2625496386 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3567018016 0 0) 0x7f0b8c128470 con 0x7f0b8c070840 2024-03-29T11:54:50.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.245+0000 7f0b81ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:54:50.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.245+0000 7f0b81ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:54:50.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.245+0000 7f0b81ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:54:50.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.245+0000 7f0b81ffb640 10 monclient(hunting): my global_id is 4487 2024-03-29T11:54:50.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.245+0000 7f0b81ffb640 1 -- 172.21.15.98:0/2625496386 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b70004eb0 con 0x7f0b8c070840 2024-03-29T11:54:50.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.245+0000 7f0b81ffb640 10 client.?.objecter ms_handle_connect 0x7f0b8c0b60c0 2024-03-29T11:54:50.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.245+0000 7f0b81ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:54:50.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.245+0000 7f0b81ffb640 1 -- 172.21.15.98:0/2625496386 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1865581076 0 0) 0x7f0b8c0ab2e0 con 0x7f0b8c0b60c0 2024-03-29T11:54:50.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.245+0000 7f0b81ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:54:50.256 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.246+0000 7f0b81ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:54:50.256 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.246+0000 7f0b81ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:54:50.256 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.246+0000 7f0b81ffb640 10 monclient(hunting): my global_id is 4474 2024-03-29T11:54:50.256 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.246+0000 7f0b81ffb640 1 -- 172.21.15.98:0/2625496386 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b8c128470 con 0x7f0b8c0b60c0 2024-03-29T11:54:50.256 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.246+0000 7f0b81ffb640 1 -- 172.21.15.98:0/2625496386 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2064282633 0 0) 0x7f0b70004eb0 con 0x7f0b8c070840 2024-03-29T11:54:50.256 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.246+0000 7f0b81ffb640 1 -- 172.21.15.98:0/2625496386 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b8c0ab2e0 con 0x7f0b8c070840 2024-03-29T11:54:50.256 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.246+0000 7f0b81ffb640 1 -- 172.21.15.98:0/2625496386 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f0b7c001a90 con 0x7f0b8c070840 2024-03-29T11:54:50.256 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.246+0000 7f0b81ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:54:50.256 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.246+0000 7f0b81ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:54:50.256 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.246+0000 7f0b81ffb640 10 monclient(hunting): dump: 2024-03-29T11:54:50.256 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:54:50.256 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:50.256 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:50.256 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:50.256 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:54:50.257 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:50.257 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:54:50.257 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:54:50.257 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:54:50.257 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:50.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.246+0000 7f0b81ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:54:50.261 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.249+0000 7f0b81ffb640 10 client.?.objecter ms_handle_connect 0x7f0b8c1dfe10 2024-03-29T11:54:50.261 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.249+0000 7f0b81ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:54:50.261 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.249+0000 7f0b81ffb640 1 -- 172.21.15.98:0/2625496386 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4188148967 0 0) 0x7f0b8c128470 con 0x7f0b8c0b60c0 2024-03-29T11:54:50.261 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.249+0000 7f0b81ffb640 1 -- 172.21.15.98:0/2625496386 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b70004eb0 con 0x7f0b8c0b60c0 2024-03-29T11:54:50.261 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.249+0000 7f0b81ffb640 1 -- 172.21.15.98:0/2625496386 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3775329465 0 0) 0x7f0b8c144040 con 0x7f0b8c1dfe10 2024-03-29T11:54:50.261 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.249+0000 7f0b81ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:54:50.261 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.249+0000 7f0b81ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:54:50.261 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.249+0000 7f0b81ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:54:50.266 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.249+0000 7f0b81ffb640 10 monclient(hunting): my global_id is 4449 2024-03-29T11:54:50.266 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.249+0000 7f0b81ffb640 1 -- 172.21.15.98:0/2625496386 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b8c128470 con 0x7f0b8c1dfe10 2024-03-29T11:54:50.266 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.249+0000 7f0b81ffb640 1 -- 172.21.15.98:0/2625496386 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2236701346 0 0) 0x7f0b8c0ab2e0 con 0x7f0b8c070840 2024-03-29T11:54:50.266 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.249+0000 7f0b81ffb640 1 -- 172.21.15.98:0/2625496386 >> v1:172.21.15.121:6789/0 conn(0x7f0b8c0b60c0 legacy=0x7f0b8c0a94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:50.266 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.249+0000 7f0b81ffb640 1 -- 172.21.15.98:0/2625496386 >> v1:172.21.15.98:6789/0 conn(0x7f0b8c1dfe10 legacy=0x7f0b8c1e2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:50.266 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.249+0000 7f0b81ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:54:50.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.249+0000 7f0b81ffb640 1 monclient: found mon.c 2024-03-29T11:54:50.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.249+0000 7f0b81ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:54:50.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.249+0000 7f0b81ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:54:50.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.249+0000 7f0b81ffb640 1 -- 172.21.15.98:0/2625496386 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b8c0ab530 con 0x7f0b8c070840 2024-03-29T11:54:50.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.249+0000 7f0b81ffb640 10 monclient: _finish_auth 0 2024-03-29T11:54:50.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.249+0000 7f0b81ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:54:50.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.249+0000 7f0b81ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:54:50.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.249+0000 7f0b938ae640 5 monclient: authenticate success, global_id 4487 2024-03-29T11:54:50.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.249+0000 7f0b938ae640 10 monclient: _renew_subs 2024-03-29T11:54:50.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.249+0000 7f0b938ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:54:50.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.249+0000 7f0b938ae640 1 -- 172.21.15.98:0/2625496386 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b8c0ab810 con 0x7f0b8c070840 2024-03-29T11:54:50.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.249+0000 7f0b938ae640 10 client.4487.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:54:50.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.249+0000 7f0b938ae640 10 monclient: _renew_subs 2024-03-29T11:54:50.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.249+0000 7f0b938ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:54:50.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.249+0000 7f0b938ae640 1 -- 172.21.15.98:0/2625496386 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b8c1304c0 con 0x7f0b8c070840 2024-03-29T11:54:50.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.249+0000 7f0b938ae640 1 librados: init done 2024-03-29T11:54:50.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.251+0000 7f0b938ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:54:50.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.251+0000 7f0b938ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:54:50.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.251+0000 7f0b938ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:54:50.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.251+0000 7f0b938ae640 1 -- 172.21.15.98:0/2625496386 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b8c130180 con 0x7f0b8c070840 2024-03-29T11:54:50.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.254+0000 7f0b81ffb640 1 -- 172.21.15.98:0/2625496386 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b7c001cd0 con 0x7f0b8c070840 2024-03-29T11:54:50.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.254+0000 7f0b81ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:54:50.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.254+0000 7f0b81ffb640 1 -- 172.21.15.98:0/2625496386 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f0b7c003a60 con 0x7f0b8c070840 2024-03-29T11:54:50.269 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.254+0000 7f0b81ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:54:50.269 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.254+0000 7f0b81ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:54:50.269 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.254+0000 7f0b81ffb640 10 monclient: dump: 2024-03-29T11:54:50.269 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:54:50.269 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:50.269 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:50.269 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:50.269 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:54:50.269 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:50.269 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:54:50.269 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:54:50.269 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:54:50.269 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:50.269 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.254+0000 7f0b81ffb640 1 -- 172.21.15.98:0/2625496386 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f0b7c003c80 con 0x7f0b8c070840 2024-03-29T11:54:50.270 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.254+0000 7f0b81ffb640 1 -- 172.21.15.98:0/2625496386 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f0b7c084bd0 con 0x7f0b8c070840 2024-03-29T11:54:50.270 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.254+0000 7f0b81ffb640 10 client.4487.objecter ms_dispatch 0x7f0b8c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:54:50.270 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.254+0000 7f0b81ffb640 3 client.4487.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:54:50.270 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.254+0000 7f0b81ffb640 3 client.4487.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:54:50.270 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.254+0000 7f0b81ffb640 20 client.4487.objecter dump_active .. 0 homeless 2024-03-29T11:54:50.270 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.255+0000 7f0b837fe640 4 set_mon_vals no callback set 2024-03-29T11:54:50.270 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.256+0000 7f0b90e49640 10 monclient: get_auth_request con 0x7f0b7006de60 auth_method 0 2024-03-29T11:54:50.270 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.256+0000 7f0b81ffb640 10 client.4487.objecter ms_handle_connect 0x7f0b7006de60 2024-03-29T11:54:50.274 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.264+0000 7f0b81ffb640 1 -- 172.21.15.98:0/2625496386 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0b7c050530 con 0x7f0b8c070840 2024-03-29T11:54:50.274 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.264+0000 7f0b81ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:54:50.274 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.264+0000 7f0b81ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:54:50.606 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.598+0000 7f0b938ae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:54:50.606 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.598+0000 7f0b938ae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:54:50.606 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.598+0000 7f0b938ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:54:50.606 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.598+0000 7f0b938ae640 1 -- 172.21.15.98:0/2625496386 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b8c0008d0 con 0x7f0b8c070840 2024-03-29T11:54:50.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.602+0000 7f0b81ffb640 1 -- 172.21.15.98:0/2625496386 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f0b7c052460 con 0x7f0b8c070840 2024-03-29T11:54:50.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.602+0000 7f0b81ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:54:50.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.602+0000 7f0b81ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:54:50.610 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:54:50.610 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:54:50.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.605+0000 7f0b938ae640 10 librados: watch_flush enter 2024-03-29T11:54:50.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.605+0000 7f0b938ae640 10 librados: watch_flush exit 2024-03-29T11:54:50.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.605+0000 7f0b938ae640 20 client.4487.objecter shutdown clearing up homeless session... 2024-03-29T11:54:50.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.605+0000 7f0b938ae640 10 client.4487.objecter successfully canceled tick 2024-03-29T11:54:50.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.606+0000 7f0b938ae640 1 -- 172.21.15.98:0/2625496386 >> v1:172.21.15.98:6816/4065186974 conn(0x7f0b7006de60 legacy=0x7f0b70070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:50.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.606+0000 7f0b938ae640 10 monclient: shutdown 2024-03-29T11:54:50.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.606+0000 7f0b938ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:54:50.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.606+0000 7f0b938ae640 1 -- 172.21.15.98:0/2625496386 >> v1:172.21.15.98:6790/0 conn(0x7f0b8c070840 legacy=0x7f0b8c140150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:50.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.609+0000 7f0b938ae640 1 -- 172.21.15.98:0/2625496386 shutdown_connections 2024-03-29T11:54:50.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.609+0000 7f0b938ae640 1 -- 172.21.15.98:0/2625496386 >> 172.21.15.98:0/2625496386 conn(0x7f0b8c067120 msgr2=0x7f0b8c06fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:54:50.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.609+0000 7f0b938ae640 1 -- 172.21.15.98:0/2625496386 shutdown_connections 2024-03-29T11:54:50.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.609+0000 7f0b938ae640 1 -- 172.21.15.98:0/2625496386 wait complete. 2024-03-29T11:54:50.619 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:50.611+0000 7f0b938ae640 1 librados: shutdown 2024-03-29T11:54:50.639 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:54:57.446 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:54:57.581 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.572+0000 7f00fac4e640 10 monclient: get_monmap_and_config 2024-03-29T11:54:57.581 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.572+0000 7f00fac4e640 10 monclient: build_initial_monmap 2024-03-29T11:54:57.581 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.572+0000 7f00fac4e640 10 monclient: monmap: 2024-03-29T11:54:57.581 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:54:57.581 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:57.581 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:54:57.573413+0000 2024-03-29T11:54:57.581 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:54:57.573413+0000 2024-03-29T11:54:57.581 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:54:57.581 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:57.581 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:54:57.581 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:54:57.582 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:54:57.582 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:57.582 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.573+0000 7f00fac4e640 1 Processor -- start 2024-03-29T11:54:57.583 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.573+0000 7f00fac4e640 1 -- start start 2024-03-29T11:54:57.583 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.573+0000 7f00fac4e640 10 monclient: init 2024-03-29T11:54:57.583 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.575+0000 7f00fac4e640 10 monclient: _reopen_session rank -1 2024-03-29T11:54:57.583 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.575+0000 7f00fac4e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:54:57.583 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.575+0000 7f00fac4e640 10 monclient(hunting): picked mon.noname-a con 0x7f00f40b4100 addr v1:172.21.15.98:6789/0 2024-03-29T11:54:57.583 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.575+0000 7f00fac4e640 10 monclient(hunting): picked mon.noname-b con 0x7f00f413ee90 addr v1:172.21.15.98:6790/0 2024-03-29T11:54:57.583 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.575+0000 7f00fac4e640 10 monclient(hunting): picked mon.noname-c con 0x7f00f40a9ac0 addr v1:172.21.15.121:6789/0 2024-03-29T11:54:57.583 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.575+0000 7f00fac4e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f40af6f0 con 0x7f00f40b4100 2024-03-29T11:54:57.583 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.575+0000 7f00fac4e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f40aac80 con 0x7f00f40a9ac0 2024-03-29T11:54:57.583 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.575+0000 7f00fac4e640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f40aaf60 con 0x7f00f413ee90 2024-03-29T11:54:57.583 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.575+0000 7f00fac4e640 10 monclient(hunting): _renew_subs 2024-03-29T11:54:57.583 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.575+0000 7f00fac4e640 10 monclient(hunting): authenticate will time out at 1204.660242s 2024-03-29T11:54:57.583 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.575+0000 7f00f3fff640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f00f413ee90 0x7f00f4146ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:35880/0 (socket says 172.21.15.98:35880) 2024-03-29T11:54:57.584 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.575+0000 7f00f3fff640 1 -- 172.21.15.98:0/3415930487 learned_addr learned my addr 172.21.15.98:0/3415930487 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:54:57.584 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.575+0000 7f00f37fe640 1 -- 172.21.15.98:0/3415930487 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3643571618 0 0) 0x7f00f40aaf60 con 0x7f00f413ee90 2024-03-29T11:54:57.584 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.575+0000 7f00f37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:54:57.584 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.575+0000 7f00f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:54:57.584 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.575+0000 7f00f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:54:57.584 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.575+0000 7f00f37fe640 10 monclient(hunting): my global_id is 4490 2024-03-29T11:54:57.584 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.575+0000 7f00f37fe640 1 -- 172.21.15.98:0/3415930487 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00e4003520 con 0x7f00f413ee90 2024-03-29T11:54:57.584 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.575+0000 7f00f37fe640 1 -- 172.21.15.98:0/3415930487 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 985720382 0 0) 0x7f00e4003520 con 0x7f00f413ee90 2024-03-29T11:54:57.584 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.575+0000 7f00f37fe640 1 -- 172.21.15.98:0/3415930487 >> v1:172.21.15.121:6789/0 conn(0x7f00f40a9ac0 legacy=0x7f00f414a210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:54:57.587 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.578+0000 7f00f37fe640 1 -- 172.21.15.98:0/3415930487 >> v1:172.21.15.98:6789/0 conn(0x7f00f40b4100 legacy=0x7f00f413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:57.587 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.578+0000 7f00f37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:54:57.587 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.578+0000 7f00f37fe640 1 monclient: found mon.noname-b 2024-03-29T11:54:57.587 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.578+0000 7f00f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:54:57.587 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.578+0000 7f00f37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:54:57.587 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.578+0000 7f00f37fe640 1 -- 172.21.15.98:0/3415930487 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f413e970 con 0x7f00f413ee90 2024-03-29T11:54:57.587 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.578+0000 7f00f37fe640 10 monclient: _finish_auth 0 2024-03-29T11:54:57.587 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.578+0000 7f00f37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:54:57.587 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.578+0000 7f00f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:54:57.587 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.578+0000 7f00f37fe640 1 -- 172.21.15.98:0/3415930487 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f00e0001690 con 0x7f00f413ee90 2024-03-29T11:54:57.587 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.578+0000 7f00f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:54:57.587 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.578+0000 7f00f37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:54:57.587 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.578+0000 7f00f37fe640 10 monclient: dump: 2024-03-29T11:54:57.587 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:54:57.588 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:57.588 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:57.588 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:57.588 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:54:57.588 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:57.588 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:54:57.588 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:54:57.588 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:54:57.588 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:57.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.578+0000 7f00fac4e640 5 monclient: authenticate success, global_id 4490 2024-03-29T11:54:57.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.578+0000 7f00fac4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:54:57.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.578+0000 7f00f37fe640 1 -- 172.21.15.98:0/3415930487 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00e0001860 con 0x7f00f413ee90 2024-03-29T11:54:57.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.578+0000 7f00f37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:54:57.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.578+0000 7f00f37fe640 1 -- 172.21.15.98:0/3415930487 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f00e0001cb0 con 0x7f00f413ee90 2024-03-29T11:54:57.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.578+0000 7f00f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:54:57.589 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.578+0000 7f00f37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:54:57.589 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.578+0000 7f00f37fe640 10 monclient: dump: 2024-03-29T11:54:57.589 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:54:57.589 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:57.589 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:57.589 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:57.589 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:54:57.589 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:57.589 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:54:57.589 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:54:57.589 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:54:57.589 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:57.589 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.578+0000 7f00fac4e640 10 monclient: get_monmap_and_config success 2024-03-29T11:54:57.589 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.578+0000 7f00fac4e640 4 set_mon_vals no callback set 2024-03-29T11:54:57.597 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.579+0000 7f00fac4e640 10 monclient: shutdown 2024-03-29T11:54:57.597 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.579+0000 7f00fac4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:54:57.597 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.579+0000 7f00fac4e640 1 -- 172.21.15.98:0/3415930487 >> v1:172.21.15.98:6790/0 conn(0x7f00f413ee90 legacy=0x7f00f4146ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:57.597 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.579+0000 7f00fac4e640 1 -- 172.21.15.98:0/3415930487 shutdown_connections 2024-03-29T11:54:57.597 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.579+0000 7f00fac4e640 1 -- 172.21.15.98:0/3415930487 >> 172.21.15.98:0/3415930487 conn(0x7f00f40b35e0 msgr2=0x7f00f40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:54:57.597 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.580+0000 7f00fac4e640 1 -- 172.21.15.98:0/3415930487 shutdown_connections 2024-03-29T11:54:57.597 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.580+0000 7f00fac4e640 1 -- 172.21.15.98:0/3415930487 wait complete. 2024-03-29T11:54:57.597 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.580+0000 7f00fac4e640 10 monclient: build_initial_monmap 2024-03-29T11:54:57.597 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.580+0000 7f00fac4e640 10 monclient: monmap: 2024-03-29T11:54:57.597 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:54:57.597 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:54:57.597 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:54:57.598 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:54:57.598 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:54:57.598 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:57.598 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:54:57.598 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:54:57.598 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:54:57.598 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:57.598 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.581+0000 7f00fac4e640 1 librados: starting msgr at 2024-03-29T11:54:57.598 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.581+0000 7f00fac4e640 1 librados: starting objecter 2024-03-29T11:54:57.598 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.581+0000 7f00fac4e640 1 Processor -- start 2024-03-29T11:54:57.598 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.581+0000 7f00fac4e640 1 -- start start 2024-03-29T11:54:57.598 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.581+0000 7f00fac4e640 1 librados: setting wanted keys 2024-03-29T11:54:57.598 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.581+0000 7f00fac4e640 1 librados: calling monclient init 2024-03-29T11:54:57.598 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.581+0000 7f00fac4e640 10 monclient: init 2024-03-29T11:54:57.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.582+0000 7f00fac4e640 10 monclient: _reopen_session rank -1 2024-03-29T11:54:57.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.582+0000 7f00fac4e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:54:57.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.582+0000 7f00fac4e640 10 monclient(hunting): picked mon.noname-c con 0x7f00f40a9ac0 addr v1:172.21.15.98:6790/0 2024-03-29T11:54:57.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.582+0000 7f00fac4e640 10 monclient(hunting): picked mon.noname-b con 0x7f00f40b4100 addr v1:172.21.15.121:6789/0 2024-03-29T11:54:57.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.582+0000 7f00fac4e640 10 monclient(hunting): picked mon.noname-a con 0x7f00f413ee90 addr v1:172.21.15.98:6789/0 2024-03-29T11:54:57.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.583+0000 7f00f91eb640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f00f413ee90 0x7f00f41ef8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37390/0 (socket says 172.21.15.98:37390) 2024-03-29T11:54:57.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.583+0000 7f00f91eb640 1 -- 172.21.15.98:0/1962398949 learned_addr learned my addr 172.21.15.98:0/1962398949 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:54:57.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.583+0000 7f00fac4e640 1 -- 172.21.15.98:0/1962398949 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f41da260 con 0x7f00f413ee90 2024-03-29T11:54:57.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.583+0000 7f00f17fa640 10 client.?.objecter ms_handle_connect 0x7f00f413ee90 2024-03-29T11:54:57.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.583+0000 7f00f17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:54:57.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.583+0000 7f00fac4e640 1 -- 172.21.15.98:0/1962398949 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f41da780 con 0x7f00f40b4100 2024-03-29T11:54:57.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.583+0000 7f00fac4e640 1 -- 172.21.15.98:0/1962398949 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f4142440 con 0x7f00f40a9ac0 2024-03-29T11:54:57.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.583+0000 7f00fac4e640 10 monclient(hunting): _renew_subs 2024-03-29T11:54:57.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.583+0000 7f00f17fa640 1 -- 172.21.15.98:0/1962398949 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2545679234 0 0) 0x7f00f41da260 con 0x7f00f413ee90 2024-03-29T11:54:57.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.583+0000 7f00fac4e640 10 monclient(hunting): authenticate will time out at 1204.668757s 2024-03-29T11:54:57.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.583+0000 7f00f17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:54:57.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.583+0000 7f00f17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:54:57.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.583+0000 7f00f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:54:57.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.583+0000 7f00f17fa640 10 monclient(hunting): my global_id is 4455 2024-03-29T11:54:57.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.583+0000 7f00f17fa640 1 -- 172.21.15.98:0/1962398949 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00d8004eb0 con 0x7f00f413ee90 2024-03-29T11:54:57.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.584+0000 7f00f17fa640 1 -- 172.21.15.98:0/1962398949 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 70298435 0 0) 0x7f00d8004eb0 con 0x7f00f413ee90 2024-03-29T11:54:57.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.584+0000 7f00f17fa640 1 -- 172.21.15.98:0/1962398949 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00f41da260 con 0x7f00f413ee90 2024-03-29T11:54:57.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.584+0000 7f00f17fa640 10 client.?.objecter ms_handle_connect 0x7f00f40a9ac0 2024-03-29T11:54:57.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.584+0000 7f00f17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:54:57.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.584+0000 7f00f17fa640 1 -- 172.21.15.98:0/1962398949 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f00ec001c60 con 0x7f00f413ee90 2024-03-29T11:54:57.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.584+0000 7f00f17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:54:57.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.584+0000 7f00f17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:54:57.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.584+0000 7f00f17fa640 10 monclient(hunting): dump: 2024-03-29T11:54:57.600 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:54:57.601 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:57.601 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:57.601 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:57.601 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:54:57.601 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:57.601 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:54:57.601 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:54:57.601 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:54:57.601 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:57.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.584+0000 7f00f17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:54:57.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.584+0000 7f00f17fa640 1 -- 172.21.15.98:0/1962398949 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3892196853 0 0) 0x7f00f4142440 con 0x7f00f40a9ac0 2024-03-29T11:54:57.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.584+0000 7f00f17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:54:57.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.584+0000 7f00f17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:54:57.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.584+0000 7f00f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:54:57.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.584+0000 7f00f17fa640 10 monclient(hunting): my global_id is 4493 2024-03-29T11:54:57.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.584+0000 7f00f17fa640 1 -- 172.21.15.98:0/1962398949 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00d8004eb0 con 0x7f00f40a9ac0 2024-03-29T11:54:57.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.585+0000 7f00f17fa640 1 -- 172.21.15.98:0/1962398949 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1607476185 0 0) 0x7f00f41da260 con 0x7f00f413ee90 2024-03-29T11:54:57.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.585+0000 7f00f17fa640 1 -- 172.21.15.98:0/1962398949 >> v1:172.21.15.98:6790/0 conn(0x7f00f40a9ac0 legacy=0x7f00f41e40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:57.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.585+0000 7f00f17fa640 1 -- 172.21.15.98:0/1962398949 >> v1:172.21.15.121:6789/0 conn(0x7f00f40b4100 legacy=0x7f00f41397f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:54:57.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.586+0000 7f00f17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:54:57.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.586+0000 7f00f17fa640 1 monclient: found mon.a 2024-03-29T11:54:57.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.586+0000 7f00f17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:54:57.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.586+0000 7f00f17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:54:57.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.586+0000 7f00f17fa640 1 -- 172.21.15.98:0/1962398949 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f41425d0 con 0x7f00f413ee90 2024-03-29T11:54:57.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.586+0000 7f00f17fa640 10 monclient: _finish_auth 0 2024-03-29T11:54:57.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.586+0000 7f00f17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:54:57.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.586+0000 7f00f17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:54:57.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.586+0000 7f00f17fa640 1 -- 172.21.15.98:0/1962398949 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00ec0020c0 con 0x7f00f413ee90 2024-03-29T11:54:57.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.586+0000 7f00f17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:54:57.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.586+0000 7f00f17fa640 1 -- 172.21.15.98:0/1962398949 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f00ec001f40 con 0x7f00f413ee90 2024-03-29T11:54:57.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.586+0000 7f00f17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:54:57.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.586+0000 7f00f17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:54:57.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.586+0000 7f00f17fa640 10 monclient: dump: 2024-03-29T11:54:57.603 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:54:57.603 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:54:57.603 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:57.603 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:54:57.603 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:54:57.603 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:54:57.603 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:54:57.603 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:54:57.603 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:54:57.603 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:54:57.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.587+0000 7f00fac4e640 5 monclient: authenticate success, global_id 4455 2024-03-29T11:54:57.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.587+0000 7f00fac4e640 10 monclient: _renew_subs 2024-03-29T11:54:57.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.587+0000 7f00fac4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:54:57.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.587+0000 7f00fac4e640 1 -- 172.21.15.98:0/1962398949 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00f4141920 con 0x7f00f413ee90 2024-03-29T11:54:57.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.587+0000 7f00f27fc640 4 set_mon_vals no callback set 2024-03-29T11:54:57.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.587+0000 7f00fac4e640 10 client.4455.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:54:57.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.587+0000 7f00fac4e640 10 monclient: _renew_subs 2024-03-29T11:54:57.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.587+0000 7f00fac4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:54:57.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.587+0000 7f00fac4e640 1 -- 172.21.15.98:0/1962398949 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f00f41e49a0 con 0x7f00f413ee90 2024-03-29T11:54:57.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.587+0000 7f00fac4e640 1 librados: init done 2024-03-29T11:54:57.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.588+0000 7f00fac4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:54:57.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.588+0000 7f00fac4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:54:57.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.588+0000 7f00fac4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:54:57.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.588+0000 7f00fac4e640 1 -- 172.21.15.98:0/1962398949 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00f40b3b50 con 0x7f00f413ee90 2024-03-29T11:54:57.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.589+0000 7f00f17fa640 1 -- 172.21.15.98:0/1962398949 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f00ec003e40 con 0x7f00f413ee90 2024-03-29T11:54:57.606 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.589+0000 7f00f17fa640 1 -- 172.21.15.98:0/1962398949 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f00ec084ed0 con 0x7f00f413ee90 2024-03-29T11:54:57.606 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.589+0000 7f00f17fa640 10 client.4455.objecter ms_dispatch 0x7f00f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:54:57.606 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.590+0000 7f00f17fa640 3 client.4455.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:54:57.606 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.590+0000 7f00f17fa640 3 client.4455.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:54:57.606 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.590+0000 7f00f89ea640 10 monclient: get_auth_request con 0x7f00d806dc70 auth_method 0 2024-03-29T11:54:57.606 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.590+0000 7f00f17fa640 20 client.4455.objecter dump_active .. 0 homeless 2024-03-29T11:54:57.606 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.590+0000 7f00f17fa640 10 client.4455.objecter ms_handle_connect 0x7f00d806dc70 2024-03-29T11:54:57.606 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.596+0000 7f00f17fa640 1 -- 172.21.15.98:0/1962398949 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f00ec050810 con 0x7f00f413ee90 2024-03-29T11:54:57.606 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.596+0000 7f00f17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:54:57.606 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.596+0000 7f00f17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:54:57.918 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.909+0000 7f00fac4e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:54:57.918 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.909+0000 7f00fac4e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:54:57.918 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.909+0000 7f00fac4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:54:57.919 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.909+0000 7f00fac4e640 1 -- 172.21.15.98:0/1962398949 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f00f41da3f0 con 0x7f00f413ee90 2024-03-29T11:54:57.919 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.909+0000 7f00f17fa640 1 -- 172.21.15.98:0/1962398949 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f00ec052740 con 0x7f00f413ee90 2024-03-29T11:54:57.919 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.909+0000 7f00f17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:54:57.919 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.909+0000 7f00f17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:54:57.919 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:54:57.919 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:54:57.920 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.912+0000 7f00fac4e640 10 librados: watch_flush enter 2024-03-29T11:54:57.923 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.913+0000 7f00fac4e640 10 librados: watch_flush exit 2024-03-29T11:54:57.923 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.913+0000 7f00fac4e640 20 client.4455.objecter shutdown clearing up homeless session... 2024-03-29T11:54:57.923 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.913+0000 7f00fac4e640 10 client.4455.objecter successfully canceled tick 2024-03-29T11:54:57.923 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.913+0000 7f00fac4e640 1 -- 172.21.15.98:0/1962398949 >> v1:172.21.15.98:6816/4065186974 conn(0x7f00d806dc70 legacy=0x7f00d8070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:57.923 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.913+0000 7f00fac4e640 10 monclient: shutdown 2024-03-29T11:54:57.924 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.913+0000 7f00fac4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:54:57.924 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.913+0000 7f00fac4e640 1 -- 172.21.15.98:0/1962398949 >> v1:172.21.15.98:6789/0 conn(0x7f00f413ee90 legacy=0x7f00f41ef8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:54:57.924 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.913+0000 7f00fac4e640 1 -- 172.21.15.98:0/1962398949 shutdown_connections 2024-03-29T11:54:57.924 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.913+0000 7f00fac4e640 1 -- 172.21.15.98:0/1962398949 >> 172.21.15.98:0/1962398949 conn(0x7f00f40b35e0 msgr2=0x7f00f413a700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:54:57.924 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.913+0000 7f00fac4e640 1 -- 172.21.15.98:0/1962398949 shutdown_connections 2024-03-29T11:54:57.924 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.913+0000 7f00fac4e640 1 -- 172.21.15.98:0/1962398949 wait complete. 2024-03-29T11:54:57.924 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:54:57.913+0000 7f00fac4e640 1 librados: shutdown 2024-03-29T11:54:57.936 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:55:04.744 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:55:04.898 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.889+0000 7f38565af640 10 monclient: get_monmap_and_config 2024-03-29T11:55:04.898 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.889+0000 7f38565af640 10 monclient: build_initial_monmap 2024-03-29T11:55:04.898 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.889+0000 7f38565af640 10 monclient: monmap: 2024-03-29T11:55:04.898 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:55:04.898 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:55:04.898 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:55:04.890744+0000 2024-03-29T11:55:04.898 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:55:04.890744+0000 2024-03-29T11:55:04.899 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:55:04.899 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:04.899 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:55:04.899 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:55:04.899 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:55:04.899 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:04.900 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.890+0000 7f38565af640 1 Processor -- start 2024-03-29T11:55:04.900 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.890+0000 7f38565af640 1 -- start start 2024-03-29T11:55:04.900 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.890+0000 7f38565af640 10 monclient: init 2024-03-29T11:55:04.900 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.891+0000 7f38565af640 10 monclient: _reopen_session rank -1 2024-03-29T11:55:04.900 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.891+0000 7f38565af640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:55:04.900 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.891+0000 7f38565af640 10 monclient(hunting): picked mon.noname-c con 0x7f38500b0770 addr v1:172.21.15.121:6789/0 2024-03-29T11:55:04.900 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.891+0000 7f38565af640 10 monclient(hunting): picked mon.noname-b con 0x7f385014a700 addr v1:172.21.15.98:6790/0 2024-03-29T11:55:04.901 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.891+0000 7f38565af640 10 monclient(hunting): picked mon.noname-a con 0x7f38500ac5a0 addr v1:172.21.15.98:6789/0 2024-03-29T11:55:04.901 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.891+0000 7f3854b4c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f38500ac5a0 0x7f3850151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:39994/0 (socket says 172.21.15.98:39994) 2024-03-29T11:55:04.901 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.891+0000 7f3854b4c640 1 -- 172.21.15.98:0/558900640 learned_addr learned my addr 172.21.15.98:0/558900640 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:55:04.901 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.891+0000 7f38565af640 1 -- 172.21.15.98:0/558900640 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38500aba70 con 0x7f38500ac5a0 2024-03-29T11:55:04.901 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.892+0000 7f38565af640 1 -- 172.21.15.98:0/558900640 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38500aecb0 con 0x7f38500b0770 2024-03-29T11:55:04.901 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.892+0000 7f38565af640 1 -- 172.21.15.98:0/558900640 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38500af150 con 0x7f385014a700 2024-03-29T11:55:04.901 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.892+0000 7f38565af640 10 monclient(hunting): _renew_subs 2024-03-29T11:55:04.901 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.892+0000 7f38565af640 10 monclient(hunting): authenticate will time out at 1211.977084s 2024-03-29T11:55:04.901 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.892+0000 7f384effd640 1 -- 172.21.15.98:0/558900640 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4167514244 0 0) 0x7f38500aba70 con 0x7f38500ac5a0 2024-03-29T11:55:04.901 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.892+0000 7f384effd640 10 monclient(hunting): _negotiate 2024-03-29T11:55:04.902 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.892+0000 7f384effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:55:04.902 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.892+0000 7f384effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:55:04.902 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.892+0000 7f384effd640 10 monclient(hunting): my global_id is 4458 2024-03-29T11:55:04.902 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.892+0000 7f384effd640 1 -- 172.21.15.98:0/558900640 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3838003520 con 0x7f38500ac5a0 2024-03-29T11:55:04.902 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.892+0000 7f384effd640 1 -- 172.21.15.98:0/558900640 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2197743401 0 0) 0x7f3838003520 con 0x7f38500ac5a0 2024-03-29T11:55:04.902 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.892+0000 7f384effd640 1 -- 172.21.15.98:0/558900640 >> v1:172.21.15.98:6790/0 conn(0x7f385014a700 legacy=0x7f38500abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:55:04.902 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.894+0000 7f384effd640 1 -- 172.21.15.98:0/558900640 >> v1:172.21.15.121:6789/0 conn(0x7f38500b0770 legacy=0x7f3850149ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:55:04.903 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.895+0000 7f384effd640 10 monclient: _finish_hunting 0 2024-03-29T11:55:04.903 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.895+0000 7f384effd640 1 monclient: found mon.noname-a 2024-03-29T11:55:04.903 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.895+0000 7f384effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:55:04.903 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.895+0000 7f384effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T11:55:04.903 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.895+0000 7f384effd640 1 -- 172.21.15.98:0/558900640 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38500ac9d0 con 0x7f38500ac5a0 2024-03-29T11:55:04.904 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.895+0000 7f384effd640 10 monclient: _finish_auth 0 2024-03-29T11:55:04.904 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.895+0000 7f384effd640 10 monclient: _check_auth_tickets 2024-03-29T11:55:04.904 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.895+0000 7f384effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:55:04.904 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.895+0000 7f384effd640 1 -- 172.21.15.98:0/558900640 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f3848001a10 con 0x7f38500ac5a0 2024-03-29T11:55:04.904 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.895+0000 7f384effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:55:04.904 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.895+0000 7f384effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:55:04.904 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.895+0000 7f384effd640 10 monclient: dump: 2024-03-29T11:55:04.904 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:55:04.904 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:55:04.904 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:04.904 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:04.904 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:55:04.904 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:04.904 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:55:04.904 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:55:04.905 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:55:04.905 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:04.905 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.895+0000 7f38565af640 5 monclient: authenticate success, global_id 4458 2024-03-29T11:55:04.905 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.895+0000 7f38565af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:55:04.905 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.895+0000 7f384effd640 1 -- 172.21.15.98:0/558900640 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3848001be0 con 0x7f38500ac5a0 2024-03-29T11:55:04.905 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.895+0000 7f384effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:55:04.906 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.895+0000 7f384effd640 1 -- 172.21.15.98:0/558900640 <== mon.0 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f3848002030 con 0x7f38500ac5a0 2024-03-29T11:55:04.906 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.895+0000 7f384effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:55:04.906 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.895+0000 7f384effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:55:04.906 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.895+0000 7f384effd640 10 monclient: dump: 2024-03-29T11:55:04.906 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:55:04.906 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:55:04.906 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:04.906 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:04.906 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:55:04.906 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:04.906 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:55:04.906 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:55:04.906 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:55:04.906 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:04.907 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.895+0000 7f38565af640 10 monclient: get_monmap_and_config success 2024-03-29T11:55:04.907 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.895+0000 7f38565af640 4 set_mon_vals no callback set 2024-03-29T11:55:04.907 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.896+0000 7f38565af640 10 monclient: shutdown 2024-03-29T11:55:04.907 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.896+0000 7f38565af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:55:04.907 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.896+0000 7f38565af640 1 -- 172.21.15.98:0/558900640 >> v1:172.21.15.98:6789/0 conn(0x7f38500ac5a0 legacy=0x7f3850151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:04.907 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.896+0000 7f38565af640 1 -- 172.21.15.98:0/558900640 shutdown_connections 2024-03-29T11:55:04.907 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.896+0000 7f38565af640 1 -- 172.21.15.98:0/558900640 >> 172.21.15.98:0/558900640 conn(0x7f385006b660 msgr2=0x7f385006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:55:04.909 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.897+0000 7f38565af640 1 -- 172.21.15.98:0/558900640 shutdown_connections 2024-03-29T11:55:04.909 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.897+0000 7f38565af640 1 -- 172.21.15.98:0/558900640 wait complete. 2024-03-29T11:55:04.909 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.897+0000 7f38565af640 10 monclient: build_initial_monmap 2024-03-29T11:55:04.909 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.897+0000 7f38565af640 10 monclient: monmap: 2024-03-29T11:55:04.909 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:55:04.909 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:55:04.909 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:55:04.909 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:55:04.909 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:55:04.909 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:04.909 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:55:04.909 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:55:04.909 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:55:04.910 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:04.910 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.898+0000 7f38565af640 1 librados: starting msgr at 2024-03-29T11:55:04.910 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.898+0000 7f38565af640 1 librados: starting objecter 2024-03-29T11:55:04.910 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.898+0000 7f38565af640 1 Processor -- start 2024-03-29T11:55:04.910 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.898+0000 7f38565af640 1 -- start start 2024-03-29T11:55:04.910 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.898+0000 7f38565af640 1 librados: setting wanted keys 2024-03-29T11:55:04.910 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.898+0000 7f38565af640 1 librados: calling monclient init 2024-03-29T11:55:04.910 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.898+0000 7f38565af640 10 monclient: init 2024-03-29T11:55:04.910 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.899+0000 7f38565af640 10 monclient: _reopen_session rank -1 2024-03-29T11:55:04.910 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.899+0000 7f38565af640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:55:04.910 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.899+0000 7f38565af640 10 monclient(hunting): picked mon.noname-c con 0x7f38500ac5a0 addr v1:172.21.15.98:6790/0 2024-03-29T11:55:04.910 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.899+0000 7f38565af640 10 monclient(hunting): picked mon.noname-a con 0x7f38500b0770 addr v1:172.21.15.98:6789/0 2024-03-29T11:55:04.910 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.899+0000 7f38565af640 10 monclient(hunting): picked mon.noname-b con 0x7f385014a700 addr v1:172.21.15.121:6789/0 2024-03-29T11:55:04.910 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.899+0000 7f38565af640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38501239d0 con 0x7f38500b0770 2024-03-29T11:55:04.911 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.899+0000 7f38565af640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3850127f70 con 0x7f385014a700 2024-03-29T11:55:04.911 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.899+0000 7f38565af640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38501281c0 con 0x7f38500ac5a0 2024-03-29T11:55:04.911 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.899+0000 7f38565af640 10 monclient(hunting): _renew_subs 2024-03-29T11:55:04.911 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.899+0000 7f38565af640 10 monclient(hunting): authenticate will time out at 1211.984902s 2024-03-29T11:55:04.914 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.901+0000 7f384f7fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f38500b0770 0x7f3850126880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:40296/0 (socket says 172.21.15.98:40296) 2024-03-29T11:55:04.914 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.901+0000 7f384f7fe640 1 -- 172.21.15.98:0/2972511792 learned_addr learned my addr 172.21.15.98:0/2972511792 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:55:04.914 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.901+0000 7f384cff9640 10 client.?.objecter ms_handle_connect 0x7f38500b0770 2024-03-29T11:55:04.914 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.901+0000 7f384cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:55:04.914 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.901+0000 7f384cff9640 1 -- 172.21.15.98:0/2972511792 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1386297711 0 0) 0x7f38501239d0 con 0x7f38500b0770 2024-03-29T11:55:04.915 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.901+0000 7f384cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:55:04.915 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.901+0000 7f384cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:55:04.915 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.901+0000 7f384cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:55:04.915 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.901+0000 7f384cff9640 10 monclient(hunting): my global_id is 4461 2024-03-29T11:55:04.915 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.902+0000 7f384cff9640 1 -- 172.21.15.98:0/2972511792 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f383c004eb0 con 0x7f38500b0770 2024-03-29T11:55:04.915 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.902+0000 7f384cff9640 1 -- 172.21.15.98:0/2972511792 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1941632630 0 0) 0x7f383c004eb0 con 0x7f38500b0770 2024-03-29T11:55:04.915 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.902+0000 7f384cff9640 1 -- 172.21.15.98:0/2972511792 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38501239d0 con 0x7f38500b0770 2024-03-29T11:55:04.915 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.902+0000 7f384cff9640 1 -- 172.21.15.98:0/2972511792 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f3844003fe0 con 0x7f38500b0770 2024-03-29T11:55:04.915 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.902+0000 7f384cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:55:04.915 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.902+0000 7f384cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:55:04.915 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.902+0000 7f384cff9640 10 monclient(hunting): dump: 2024-03-29T11:55:04.915 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:55:04.915 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:55:04.915 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:04.915 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:04.916 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:55:04.916 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:04.916 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:55:04.916 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:55:04.916 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:55:04.916 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:04.916 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.902+0000 7f384cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:55:04.916 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.902+0000 7f384cff9640 1 -- 172.21.15.98:0/2972511792 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3368910197 0 0) 0x7f38501239d0 con 0x7f38500b0770 2024-03-29T11:55:04.916 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.902+0000 7f384cff9640 1 -- 172.21.15.98:0/2972511792 >> v1:172.21.15.98:6790/0 conn(0x7f38500ac5a0 legacy=0x7f3850144fd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:55:04.916 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.904+0000 7f384cff9640 1 -- 172.21.15.98:0/2972511792 >> v1:172.21.15.121:6789/0 conn(0x7f385014a700 legacy=0x7f38501e6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:04.916 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.904+0000 7f384cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:55:04.916 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.904+0000 7f384cff9640 1 monclient: found mon.a 2024-03-29T11:55:04.916 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.904+0000 7f384cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:55:04.916 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.904+0000 7f384cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:55:04.917 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.904+0000 7f384cff9640 1 -- 172.21.15.98:0/2972511792 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3850126360 con 0x7f38500b0770 2024-03-29T11:55:04.917 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.904+0000 7f384cff9640 10 monclient: _finish_auth 0 2024-03-29T11:55:04.917 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.904+0000 7f384cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:55:04.917 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.904+0000 7f384cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:55:04.917 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.904+0000 7f38565af640 5 monclient: authenticate success, global_id 4461 2024-03-29T11:55:04.917 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.904+0000 7f38565af640 10 monclient: _renew_subs 2024-03-29T11:55:04.917 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.904+0000 7f38565af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:55:04.917 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.904+0000 7f38565af640 1 -- 172.21.15.98:0/2972511792 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38501dc1e0 con 0x7f38500b0770 2024-03-29T11:55:04.917 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.904+0000 7f38565af640 10 client.4461.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:55:04.917 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.904+0000 7f38565af640 10 monclient: _renew_subs 2024-03-29T11:55:04.917 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.904+0000 7f38565af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:55:04.917 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.904+0000 7f38565af640 1 -- 172.21.15.98:0/2972511792 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3850126020 con 0x7f38500b0770 2024-03-29T11:55:04.917 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.904+0000 7f38565af640 1 librados: init done 2024-03-29T11:55:04.917 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.905+0000 7f384cff9640 1 -- 172.21.15.98:0/2972511792 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3844000ed0 con 0x7f38500b0770 2024-03-29T11:55:04.917 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.905+0000 7f384cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:55:04.918 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.905+0000 7f384cff9640 1 -- 172.21.15.98:0/2972511792 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f38440012a0 con 0x7f38500b0770 2024-03-29T11:55:04.918 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.905+0000 7f384cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:55:04.918 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.906+0000 7f384cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:55:04.918 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.906+0000 7f384cff9640 10 monclient: dump: 2024-03-29T11:55:04.918 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:55:04.918 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:55:04.918 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:04.918 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:04.918 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:55:04.918 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:04.918 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:55:04.918 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:55:04.918 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:55:04.918 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:04.920 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.906+0000 7f384dffb640 4 set_mon_vals no callback set 2024-03-29T11:55:04.920 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.906+0000 7f38565af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:55:04.920 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.907+0000 7f38565af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:55:04.920 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.907+0000 7f38565af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:55:04.920 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.907+0000 7f38565af640 1 -- 172.21.15.98:0/2972511792 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3830005080 con 0x7f38500b0770 2024-03-29T11:55:04.920 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.909+0000 7f384cff9640 1 -- 172.21.15.98:0/2972511792 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f3844001400 con 0x7f38500b0770 2024-03-29T11:55:04.920 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.909+0000 7f384cff9640 1 -- 172.21.15.98:0/2972511792 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f3844084aa0 con 0x7f38500b0770 2024-03-29T11:55:04.920 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.909+0000 7f384cff9640 10 client.4461.objecter ms_dispatch 0x7f3850000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:55:04.920 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.909+0000 7f384cff9640 3 client.4461.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:55:04.921 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.909+0000 7f384cff9640 3 client.4461.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:55:04.921 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.910+0000 7f384cff9640 20 client.4461.objecter dump_active .. 0 homeless 2024-03-29T11:55:04.921 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.911+0000 7f384ffff640 10 monclient: get_auth_request con 0x7f383c06db90 auth_method 0 2024-03-29T11:55:04.921 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.911+0000 7f384cff9640 10 client.4461.objecter ms_handle_connect 0x7f383c06db90 2024-03-29T11:55:04.925 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.917+0000 7f384cff9640 1 -- 172.21.15.98:0/2972511792 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3844050380 con 0x7f38500b0770 2024-03-29T11:55:04.925 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.917+0000 7f384cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:55:04.925 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:04.917+0000 7f384cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:55:05.252 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:05.243+0000 7f38565af640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:55:05.252 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:05.244+0000 7f38565af640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:55:05.252 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:05.244+0000 7f38565af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:55:05.252 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:05.244+0000 7f38565af640 1 -- 172.21.15.98:0/2972511792 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3830005330 con 0x7f38500b0770 2024-03-29T11:55:05.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:05.245+0000 7f384cff9640 1 -- 172.21.15.98:0/2972511792 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f38440522b0 con 0x7f38500b0770 2024-03-29T11:55:05.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:05.245+0000 7f384cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:55:05.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:05.245+0000 7f384cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:55:05.253 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:55:05.253 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:55:05.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:05.249+0000 7f38565af640 10 librados: watch_flush enter 2024-03-29T11:55:05.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:05.249+0000 7f38565af640 10 librados: watch_flush exit 2024-03-29T11:55:05.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:05.249+0000 7f38565af640 20 client.4461.objecter shutdown clearing up homeless session... 2024-03-29T11:55:05.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:05.249+0000 7f38565af640 10 client.4461.objecter successfully canceled tick 2024-03-29T11:55:05.258 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:05.250+0000 7f38565af640 1 -- 172.21.15.98:0/2972511792 >> v1:172.21.15.98:6816/4065186974 conn(0x7f383c06db90 legacy=0x7f383c070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:05.262 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:05.250+0000 7f38565af640 10 monclient: shutdown 2024-03-29T11:55:05.262 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:05.250+0000 7f38565af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:55:05.262 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:05.250+0000 7f38565af640 1 -- 172.21.15.98:0/2972511792 >> v1:172.21.15.98:6789/0 conn(0x7f38500b0770 legacy=0x7f3850126880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:05.262 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:05.250+0000 7f38565af640 1 -- 172.21.15.98:0/2972511792 shutdown_connections 2024-03-29T11:55:05.262 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:05.250+0000 7f38565af640 1 -- 172.21.15.98:0/2972511792 >> 172.21.15.98:0/2972511792 conn(0x7f385006b660 msgr2=0x7f38500b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:55:05.263 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:05.254+0000 7f38565af640 1 -- 172.21.15.98:0/2972511792 shutdown_connections 2024-03-29T11:55:05.263 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:05.254+0000 7f38565af640 1 -- 172.21.15.98:0/2972511792 wait complete. 2024-03-29T11:55:05.263 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:05.254+0000 7f38565af640 1 librados: shutdown 2024-03-29T11:55:05.278 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:55:12.086 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:55:12.222 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.213+0000 7f44f0d96640 10 monclient: get_monmap_and_config 2024-03-29T11:55:12.222 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.213+0000 7f44f0d96640 10 monclient: build_initial_monmap 2024-03-29T11:55:12.223 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.213+0000 7f44f0d96640 10 monclient: monmap: 2024-03-29T11:55:12.223 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:55:12.223 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:55:12.223 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:55:12.214844+0000 2024-03-29T11:55:12.223 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:55:12.214844+0000 2024-03-29T11:55:12.223 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:55:12.223 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:12.223 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:55:12.223 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:55:12.223 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:55:12.223 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:12.226 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.215+0000 7f44f0d96640 1 Processor -- start 2024-03-29T11:55:12.226 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.215+0000 7f44f0d96640 1 -- start start 2024-03-29T11:55:12.226 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.215+0000 7f44f0d96640 10 monclient: init 2024-03-29T11:55:12.226 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.216+0000 7f44f0d96640 10 monclient: _reopen_session rank -1 2024-03-29T11:55:12.226 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.216+0000 7f44f0d96640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:55:12.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.216+0000 7f44f0d96640 10 monclient(hunting): picked mon.noname-b con 0x7f44ec0b0770 addr v1:172.21.15.98:6790/0 2024-03-29T11:55:12.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.216+0000 7f44f0d96640 10 monclient(hunting): picked mon.noname-c con 0x7f44ec14a700 addr v1:172.21.15.121:6789/0 2024-03-29T11:55:12.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.216+0000 7f44f0d96640 10 monclient(hunting): picked mon.noname-a con 0x7f44ec0ac5a0 addr v1:172.21.15.98:6789/0 2024-03-29T11:55:12.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.216+0000 7f44f0d96640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44ec0aba70 con 0x7f44ec0ac5a0 2024-03-29T11:55:12.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.216+0000 7f44f0d96640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44ec0aecb0 con 0x7f44ec14a700 2024-03-29T11:55:12.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.216+0000 7f44f0d96640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44ec0af150 con 0x7f44ec0b0770 2024-03-29T11:55:12.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.216+0000 7f44f0d96640 10 monclient(hunting): _renew_subs 2024-03-29T11:55:12.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.216+0000 7f44f0d96640 10 monclient(hunting): authenticate will time out at 1219.301491s 2024-03-29T11:55:12.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.216+0000 7f44ea59c640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f44ec0b0770 0x7f44ec149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:49726/0 (socket says 172.21.15.98:49726) 2024-03-29T11:55:12.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.216+0000 7f44ea59c640 1 -- 172.21.15.98:0/3903006884 learned_addr learned my addr 172.21.15.98:0/3903006884 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:55:12.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.216+0000 7f44e959a640 1 -- 172.21.15.98:0/3903006884 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 94203999 0 0) 0x7f44ec0af150 con 0x7f44ec0b0770 2024-03-29T11:55:12.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.216+0000 7f44e959a640 10 monclient(hunting): _negotiate 2024-03-29T11:55:12.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.216+0000 7f44e959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:55:12.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.216+0000 7f44e959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:55:12.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.216+0000 7f44e959a640 10 monclient(hunting): my global_id is 4496 2024-03-29T11:55:12.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.216+0000 7f44e959a640 1 -- 172.21.15.98:0/3903006884 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44d4003520 con 0x7f44ec0b0770 2024-03-29T11:55:12.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.217+0000 7f44e959a640 1 -- 172.21.15.98:0/3903006884 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1378268579 0 0) 0x7f44ec0aecb0 con 0x7f44ec14a700 2024-03-29T11:55:12.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.217+0000 7f44e959a640 10 monclient(hunting): _negotiate 2024-03-29T11:55:12.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.217+0000 7f44e959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:55:12.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.217+0000 7f44e959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:55:12.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.217+0000 7f44e959a640 10 monclient(hunting): my global_id is 4477 2024-03-29T11:55:12.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.217+0000 7f44e959a640 1 -- 172.21.15.98:0/3903006884 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44ec0af150 con 0x7f44ec14a700 2024-03-29T11:55:12.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.217+0000 7f44e959a640 1 -- 172.21.15.98:0/3903006884 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3208205509 0 0) 0x7f44d4003520 con 0x7f44ec0b0770 2024-03-29T11:55:12.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.217+0000 7f44e959a640 1 -- 172.21.15.98:0/3903006884 >> v1:172.21.15.121:6789/0 conn(0x7f44ec14a700 legacy=0x7f44ec0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:12.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.217+0000 7f44e959a640 1 -- 172.21.15.98:0/3903006884 >> v1:172.21.15.98:6789/0 conn(0x7f44ec0ac5a0 legacy=0x7f44ec151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:55:12.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.219+0000 7f44e959a640 10 monclient: _finish_hunting 0 2024-03-29T11:55:12.230 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.219+0000 7f44e959a640 1 monclient: found mon.noname-b 2024-03-29T11:55:12.230 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.219+0000 7f44e959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:55:12.230 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.219+0000 7f44e959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:55:12.230 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.219+0000 7f44e959a640 1 -- 172.21.15.98:0/3903006884 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44ec0ac9d0 con 0x7f44ec0b0770 2024-03-29T11:55:12.230 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.219+0000 7f44e959a640 10 monclient: _finish_auth 0 2024-03-29T11:55:12.230 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.219+0000 7f44e959a640 10 monclient: _check_auth_tickets 2024-03-29T11:55:12.230 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.219+0000 7f44e959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:55:12.230 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.219+0000 7f44f0d96640 5 monclient: authenticate success, global_id 4496 2024-03-29T11:55:12.230 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.219+0000 7f44e959a640 1 -- 172.21.15.98:0/3903006884 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f44e0001690 con 0x7f44ec0b0770 2024-03-29T11:55:12.230 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.219+0000 7f44e959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:55:12.230 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.219+0000 7f44e959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:55:12.230 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.219+0000 7f44e959a640 10 monclient: dump: 2024-03-29T11:55:12.230 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:55:12.231 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:55:12.231 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:12.231 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:12.231 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:55:12.231 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:12.231 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:55:12.231 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:55:12.231 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:55:12.231 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:12.231 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.219+0000 7f44f0d96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:55:12.231 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.219+0000 7f44e959a640 1 -- 172.21.15.98:0/3903006884 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44e00017f0 con 0x7f44ec0b0770 2024-03-29T11:55:12.231 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.219+0000 7f44e959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:55:12.231 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.219+0000 7f44e959a640 1 -- 172.21.15.98:0/3903006884 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f44e0001c00 con 0x7f44ec0b0770 2024-03-29T11:55:12.231 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.219+0000 7f44f0d96640 10 monclient: get_monmap_and_config success 2024-03-29T11:55:12.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.219+0000 7f44f0d96640 4 set_mon_vals no callback set 2024-03-29T11:55:12.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.219+0000 7f44e959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:55:12.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.219+0000 7f44e959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:55:12.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.219+0000 7f44e959a640 10 monclient: dump: 2024-03-29T11:55:12.232 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:55:12.232 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:55:12.232 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:12.232 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:12.232 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:55:12.232 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:12.232 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:55:12.232 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:55:12.232 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:55:12.232 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:12.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.220+0000 7f44f0d96640 10 monclient: shutdown 2024-03-29T11:55:12.233 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.220+0000 7f44f0d96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:55:12.233 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.220+0000 7f44f0d96640 1 -- 172.21.15.98:0/3903006884 >> v1:172.21.15.98:6790/0 conn(0x7f44ec0b0770 legacy=0x7f44ec149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:12.233 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.221+0000 7f44f0d96640 1 -- 172.21.15.98:0/3903006884 shutdown_connections 2024-03-29T11:55:12.233 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.221+0000 7f44f0d96640 1 -- 172.21.15.98:0/3903006884 >> 172.21.15.98:0/3903006884 conn(0x7f44ec06b660 msgr2=0x7f44ec06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:55:12.233 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.221+0000 7f44f0d96640 1 -- 172.21.15.98:0/3903006884 shutdown_connections 2024-03-29T11:55:12.233 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.221+0000 7f44f0d96640 1 -- 172.21.15.98:0/3903006884 wait complete. 2024-03-29T11:55:12.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.222+0000 7f44f0d96640 10 monclient: build_initial_monmap 2024-03-29T11:55:12.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.222+0000 7f44f0d96640 10 monclient: monmap: 2024-03-29T11:55:12.237 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:55:12.237 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:55:12.237 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:55:12.237 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:55:12.237 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:55:12.237 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:12.237 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:55:12.237 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:55:12.237 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:55:12.237 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:12.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.222+0000 7f44f0d96640 1 librados: starting msgr at 2024-03-29T11:55:12.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.222+0000 7f44f0d96640 1 librados: starting objecter 2024-03-29T11:55:12.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.223+0000 7f44f0d96640 1 Processor -- start 2024-03-29T11:55:12.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.223+0000 7f44f0d96640 1 -- start start 2024-03-29T11:55:12.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.223+0000 7f44f0d96640 1 librados: setting wanted keys 2024-03-29T11:55:12.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.223+0000 7f44f0d96640 1 librados: calling monclient init 2024-03-29T11:55:12.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.223+0000 7f44f0d96640 10 monclient: init 2024-03-29T11:55:12.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.223+0000 7f44f0d96640 10 monclient: _reopen_session rank -1 2024-03-29T11:55:12.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.223+0000 7f44f0d96640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:55:12.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.223+0000 7f44f0d96640 10 monclient(hunting): picked mon.noname-b con 0x7f44ec0ac5a0 addr v1:172.21.15.121:6789/0 2024-03-29T11:55:12.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.223+0000 7f44f0d96640 10 monclient(hunting): picked mon.noname-a con 0x7f44ec0b0770 addr v1:172.21.15.98:6789/0 2024-03-29T11:55:12.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.223+0000 7f44f0d96640 10 monclient(hunting): picked mon.noname-c con 0x7f44ec14a700 addr v1:172.21.15.98:6790/0 2024-03-29T11:55:12.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.223+0000 7f44f0d96640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44ec128440 con 0x7f44ec0b0770 2024-03-29T11:55:12.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.223+0000 7f44f0d96640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44ec125fe0 con 0x7f44ec0ac5a0 2024-03-29T11:55:12.239 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.223+0000 7f44f0d96640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44ec1263c0 con 0x7f44ec14a700 2024-03-29T11:55:12.239 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.223+0000 7f44f0d96640 10 monclient(hunting): _renew_subs 2024-03-29T11:55:12.239 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.223+0000 7f44f0d96640 10 monclient(hunting): authenticate will time out at 1219.308882s 2024-03-29T11:55:12.239 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.224+0000 7f44ead9d640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f44ec14a700 0x7f44ec1e7040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:55700/0 (socket says 172.21.15.98:55700) 2024-03-29T11:55:12.239 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.224+0000 7f44ead9d640 1 -- 172.21.15.98:0/3919190783 learned_addr learned my addr 172.21.15.98:0/3919190783 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:55:12.239 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.224+0000 7f44daffd640 10 client.?.objecter ms_handle_connect 0x7f44ec14a700 2024-03-29T11:55:12.239 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.224+0000 7f44daffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:55:12.239 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.225+0000 7f44daffd640 1 -- 172.21.15.98:0/3919190783 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3430871828 0 0) 0x7f44ec1263c0 con 0x7f44ec14a700 2024-03-29T11:55:12.239 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.225+0000 7f44daffd640 10 monclient(hunting): _negotiate 2024-03-29T11:55:12.239 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.225+0000 7f44daffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:55:12.239 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.225+0000 7f44daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:55:12.239 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.225+0000 7f44daffd640 10 monclient(hunting): my global_id is 4499 2024-03-29T11:55:12.239 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.225+0000 7f44daffd640 1 -- 172.21.15.98:0/3919190783 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44cc004eb0 con 0x7f44ec14a700 2024-03-29T11:55:12.239 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.225+0000 7f44daffd640 1 -- 172.21.15.98:0/3919190783 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1630056571 0 0) 0x7f44cc004eb0 con 0x7f44ec14a700 2024-03-29T11:55:12.239 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.225+0000 7f44daffd640 1 -- 172.21.15.98:0/3919190783 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44ec1263c0 con 0x7f44ec14a700 2024-03-29T11:55:12.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.225+0000 7f44daffd640 10 client.?.objecter ms_handle_connect 0x7f44ec0ac5a0 2024-03-29T11:55:12.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.225+0000 7f44daffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:55:12.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.225+0000 7f44daffd640 1 -- 172.21.15.98:0/3919190783 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f44e4004000 con 0x7f44ec14a700 2024-03-29T11:55:12.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.225+0000 7f44daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:55:12.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.225+0000 7f44daffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:55:12.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.225+0000 7f44daffd640 10 monclient(hunting): dump: 2024-03-29T11:55:12.240 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:55:12.240 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:55:12.240 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:12.240 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:12.240 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:55:12.240 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:12.240 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:55:12.240 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:55:12.240 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:55:12.241 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:12.241 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.225+0000 7f44daffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:55:12.241 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.226+0000 7f44daffd640 1 -- 172.21.15.98:0/3919190783 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 97263739 0 0) 0x7f44ec125fe0 con 0x7f44ec0ac5a0 2024-03-29T11:55:12.241 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.226+0000 7f44daffd640 10 monclient(hunting): _negotiate 2024-03-29T11:55:12.241 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.226+0000 7f44daffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:55:12.241 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.226+0000 7f44daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:55:12.241 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.226+0000 7f44daffd640 10 monclient(hunting): my global_id is 4480 2024-03-29T11:55:12.241 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.226+0000 7f44daffd640 1 -- 172.21.15.98:0/3919190783 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44cc004eb0 con 0x7f44ec0ac5a0 2024-03-29T11:55:12.241 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.226+0000 7f44daffd640 1 -- 172.21.15.98:0/3919190783 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 674285931 0 0) 0x7f44ec1263c0 con 0x7f44ec14a700 2024-03-29T11:55:12.241 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.226+0000 7f44daffd640 1 -- 172.21.15.98:0/3919190783 >> v1:172.21.15.121:6789/0 conn(0x7f44ec0ac5a0 legacy=0x7f44ec1268e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:12.241 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.226+0000 7f44daffd640 1 -- 172.21.15.98:0/3919190783 >> v1:172.21.15.98:6789/0 conn(0x7f44ec0b0770 legacy=0x7f44ec144fc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:55:12.241 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.227+0000 7f44daffd640 10 monclient: _finish_hunting 0 2024-03-29T11:55:12.241 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.227+0000 7f44daffd640 1 monclient: found mon.c 2024-03-29T11:55:12.242 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.227+0000 7f44daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:55:12.242 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.227+0000 7f44daffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:55:12.242 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.227+0000 7f44daffd640 1 -- 172.21.15.98:0/3919190783 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44ec127440 con 0x7f44ec14a700 2024-03-29T11:55:12.242 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.227+0000 7f44daffd640 10 monclient: _finish_auth 0 2024-03-29T11:55:12.242 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.227+0000 7f44daffd640 10 monclient: _check_auth_tickets 2024-03-29T11:55:12.242 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.227+0000 7f44daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:55:12.242 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.227+0000 7f44f0d96640 5 monclient: authenticate success, global_id 4499 2024-03-29T11:55:12.242 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.227+0000 7f44f0d96640 10 monclient: _renew_subs 2024-03-29T11:55:12.242 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.227+0000 7f44f0d96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:55:12.242 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.227+0000 7f44f0d96640 1 -- 172.21.15.98:0/3919190783 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44ec1239d0 con 0x7f44ec14a700 2024-03-29T11:55:12.242 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.227+0000 7f44f0d96640 10 client.4499.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:55:12.242 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.227+0000 7f44f0d96640 10 monclient: _renew_subs 2024-03-29T11:55:12.243 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.227+0000 7f44f0d96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:55:12.243 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.227+0000 7f44f0d96640 1 -- 172.21.15.98:0/3919190783 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f44ec1271d0 con 0x7f44ec14a700 2024-03-29T11:55:12.243 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.227+0000 7f44f0d96640 1 librados: init done 2024-03-29T11:55:12.243 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.227+0000 7f44daffd640 1 -- 172.21.15.98:0/3919190783 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44e4004160 con 0x7f44ec14a700 2024-03-29T11:55:12.243 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.227+0000 7f44daffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:55:12.243 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.227+0000 7f44daffd640 1 -- 172.21.15.98:0/3919190783 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f44e40010f0 con 0x7f44ec14a700 2024-03-29T11:55:12.243 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.227+0000 7f44daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:55:12.243 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.227+0000 7f44daffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:55:12.243 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.227+0000 7f44e8d99640 4 set_mon_vals no callback set 2024-03-29T11:55:12.243 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.227+0000 7f44daffd640 10 monclient: dump: 2024-03-29T11:55:12.243 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:55:12.243 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:55:12.243 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:12.244 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:12.244 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:55:12.244 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:12.244 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:55:12.244 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:55:12.244 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:55:12.244 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:12.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.228+0000 7f44f0d96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:55:12.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.228+0000 7f44f0d96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:55:12.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.228+0000 7f44f0d96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:55:12.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.229+0000 7f44f0d96640 1 -- 172.21.15.98:0/3919190783 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44d0005080 con 0x7f44ec14a700 2024-03-29T11:55:12.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.229+0000 7f44daffd640 1 -- 172.21.15.98:0/3919190783 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f44e4001250 con 0x7f44ec14a700 2024-03-29T11:55:12.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.230+0000 7f44daffd640 1 -- 172.21.15.98:0/3919190783 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f44e40849f0 con 0x7f44ec14a700 2024-03-29T11:55:12.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.230+0000 7f44daffd640 10 client.4499.objecter ms_dispatch 0x7f44ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:55:12.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.230+0000 7f44daffd640 3 client.4499.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:55:12.246 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.230+0000 7f44daffd640 3 client.4499.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:55:12.246 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.230+0000 7f44daffd640 20 client.4499.objecter dump_active .. 0 homeless 2024-03-29T11:55:12.246 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.230+0000 7f44ea59c640 10 monclient: get_auth_request con 0x7f44cc06dc40 auth_method 0 2024-03-29T11:55:12.246 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.230+0000 7f44daffd640 10 client.4499.objecter ms_handle_connect 0x7f44cc06dc40 2024-03-29T11:55:12.246 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.235+0000 7f44daffd640 1 -- 172.21.15.98:0/3919190783 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f44e4050100 con 0x7f44ec14a700 2024-03-29T11:55:12.246 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.235+0000 7f44daffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:55:12.246 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.235+0000 7f44daffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:55:12.551 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.543+0000 7f44f0d96640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:55:12.551 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.543+0000 7f44f0d96640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:55:12.552 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.543+0000 7f44f0d96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:55:12.552 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.543+0000 7f44f0d96640 1 -- 172.21.15.98:0/3919190783 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44d0005330 con 0x7f44ec14a700 2024-03-29T11:55:12.552 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.544+0000 7f44daffd640 1 -- 172.21.15.98:0/3919190783 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f44e4052200 con 0x7f44ec14a700 2024-03-29T11:55:12.552 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.544+0000 7f44daffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:55:12.552 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.544+0000 7f44daffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:55:12.553 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:55:12.553 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:55:12.555 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.546+0000 7f44f0d96640 10 librados: watch_flush enter 2024-03-29T11:55:12.555 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.547+0000 7f44f0d96640 10 librados: watch_flush exit 2024-03-29T11:55:12.555 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.547+0000 7f44f0d96640 20 client.4499.objecter shutdown clearing up homeless session... 2024-03-29T11:55:12.555 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.547+0000 7f44f0d96640 10 client.4499.objecter successfully canceled tick 2024-03-29T11:55:12.555 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.547+0000 7f44f0d96640 1 -- 172.21.15.98:0/3919190783 >> v1:172.21.15.98:6816/4065186974 conn(0x7f44cc06dc40 legacy=0x7f44cc070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:12.555 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.547+0000 7f44f0d96640 10 monclient: shutdown 2024-03-29T11:55:12.556 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.547+0000 7f44f0d96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:55:12.556 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.547+0000 7f44f0d96640 1 -- 172.21.15.98:0/3919190783 >> v1:172.21.15.98:6790/0 conn(0x7f44ec14a700 legacy=0x7f44ec1e7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:12.556 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.547+0000 7f44f0d96640 1 -- 172.21.15.98:0/3919190783 shutdown_connections 2024-03-29T11:55:12.556 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.547+0000 7f44f0d96640 1 -- 172.21.15.98:0/3919190783 >> 172.21.15.98:0/3919190783 conn(0x7f44ec06b660 msgr2=0x7f44ec06ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:55:12.556 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.547+0000 7f44f0d96640 1 -- 172.21.15.98:0/3919190783 shutdown_connections 2024-03-29T11:55:12.556 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.547+0000 7f44f0d96640 1 -- 172.21.15.98:0/3919190783 wait complete. 2024-03-29T11:55:12.556 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:12.547+0000 7f44f0d96640 1 librados: shutdown 2024-03-29T11:55:12.569 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:55:19.377 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:55:19.533 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.522+0000 7f91b804e640 10 monclient: get_monmap_and_config 2024-03-29T11:55:19.533 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.522+0000 7f91b804e640 10 monclient: build_initial_monmap 2024-03-29T11:55:19.533 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.522+0000 7f91b804e640 10 monclient: monmap: 2024-03-29T11:55:19.533 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:55:19.533 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:55:19.533 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:55:19.523669+0000 2024-03-29T11:55:19.533 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:55:19.523669+0000 2024-03-29T11:55:19.534 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:55:19.534 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:19.534 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:55:19.534 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:55:19.534 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:55:19.534 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:19.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.523+0000 7f91b804e640 1 Processor -- start 2024-03-29T11:55:19.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.524+0000 7f91b804e640 1 -- start start 2024-03-29T11:55:19.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.524+0000 7f91b804e640 10 monclient: init 2024-03-29T11:55:19.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.525+0000 7f91b804e640 10 monclient: _reopen_session rank -1 2024-03-29T11:55:19.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.525+0000 7f91b804e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:55:19.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.525+0000 7f91b804e640 10 monclient(hunting): picked mon.noname-b con 0x7f91b00ad6c0 addr v1:172.21.15.98:6790/0 2024-03-29T11:55:19.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.525+0000 7f91b804e640 10 monclient(hunting): picked mon.noname-c con 0x7f91b0152980 addr v1:172.21.15.121:6789/0 2024-03-29T11:55:19.535 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.525+0000 7f91b804e640 10 monclient(hunting): picked mon.noname-a con 0x7f91b01588d0 addr v1:172.21.15.98:6789/0 2024-03-29T11:55:19.535 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.525+0000 7f91b804e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91b00a94e0 con 0x7f91b01588d0 2024-03-29T11:55:19.535 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.525+0000 7f91b804e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91b00af3a0 con 0x7f91b0152980 2024-03-29T11:55:19.535 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.525+0000 7f91b804e640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91b00afb60 con 0x7f91b00ad6c0 2024-03-29T11:55:19.535 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.525+0000 7f91b804e640 10 monclient(hunting): _renew_subs 2024-03-29T11:55:19.535 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.525+0000 7f91b804e640 10 monclient(hunting): authenticate will time out at 1226.610396s 2024-03-29T11:55:19.536 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.528+0000 7f91b55e9640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f91b0152980 0x7f91b0152df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:35083/0 (socket says 172.21.15.98:35083) 2024-03-29T11:55:19.537 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.528+0000 7f91b55e9640 1 -- 172.21.15.98:0/2476545554 learned_addr learned my addr 172.21.15.98:0/2476545554 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:55:19.538 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.530+0000 7f91b4de8640 1 -- 172.21.15.98:0/2476545554 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 701038054 0 0) 0x7f91b00af3a0 con 0x7f91b0152980 2024-03-29T11:55:19.538 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.530+0000 7f91b4de8640 10 monclient(hunting): _negotiate 2024-03-29T11:55:19.538 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.530+0000 7f91b4de8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:55:19.538 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.530+0000 7f91b4de8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:55:19.538 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.530+0000 7f91b4de8640 10 monclient(hunting): my global_id is 4483 2024-03-29T11:55:19.538 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.530+0000 7f91b4de8640 1 -- 172.21.15.98:0/2476545554 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f919c003520 con 0x7f91b0152980 2024-03-29T11:55:19.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.531+0000 7f91b4de8640 1 -- 172.21.15.98:0/2476545554 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 864263257 0 0) 0x7f919c003520 con 0x7f91b0152980 2024-03-29T11:55:19.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.531+0000 7f91b4de8640 1 -- 172.21.15.98:0/2476545554 >> v1:172.21.15.98:6790/0 conn(0x7f91b00ad6c0 legacy=0x7f91b014fb00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:55:19.540 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.531+0000 7f91b4de8640 1 -- 172.21.15.98:0/2476545554 >> v1:172.21.15.98:6789/0 conn(0x7f91b01588d0 legacy=0x7f91b015acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:55:19.540 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.531+0000 7f91b4de8640 10 monclient: _finish_hunting 0 2024-03-29T11:55:19.540 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.531+0000 7f91b4de8640 1 monclient: found mon.noname-c 2024-03-29T11:55:19.540 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.531+0000 7f91b4de8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:55:19.540 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.531+0000 7f91b4de8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:55:19.540 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.531+0000 7f91b4de8640 1 -- 172.21.15.98:0/2476545554 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b00aab80 con 0x7f91b0152980 2024-03-29T11:55:19.540 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.531+0000 7f91b4de8640 10 monclient: _finish_auth 0 2024-03-29T11:55:19.540 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.531+0000 7f91b4de8640 10 monclient: _check_auth_tickets 2024-03-29T11:55:19.540 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.531+0000 7f91b4de8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:55:19.540 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.531+0000 7f91b4de8640 1 -- 172.21.15.98:0/2476545554 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f91a4001690 con 0x7f91b0152980 2024-03-29T11:55:19.541 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.531+0000 7f91b804e640 5 monclient: authenticate success, global_id 4483 2024-03-29T11:55:19.541 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.531+0000 7f91b804e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:55:19.541 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.531+0000 7f91b4de8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:55:19.541 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.531+0000 7f91b4de8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:55:19.541 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.531+0000 7f91b4de8640 10 monclient: dump: 2024-03-29T11:55:19.541 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:55:19.541 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:55:19.541 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:19.541 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:19.541 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:55:19.541 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:19.541 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:55:19.541 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:55:19.542 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:55:19.542 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:19.542 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.531+0000 7f91b804e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:55:19.542 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.532+0000 7f91b4de8640 1 -- 172.21.15.98:0/2476545554 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91a4001920 con 0x7f91b0152980 2024-03-29T11:55:19.542 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.532+0000 7f91b4de8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:55:19.542 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.532+0000 7f91b4de8640 1 -- 172.21.15.98:0/2476545554 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f91a4001cb0 con 0x7f91b0152980 2024-03-29T11:55:19.542 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.532+0000 7f91b804e640 10 monclient: get_monmap_and_config success 2024-03-29T11:55:19.542 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.532+0000 7f91b804e640 4 set_mon_vals no callback set 2024-03-29T11:55:19.542 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.532+0000 7f91b4de8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:55:19.542 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.532+0000 7f91b4de8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:55:19.542 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.532+0000 7f91b4de8640 10 monclient: dump: 2024-03-29T11:55:19.542 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:55:19.542 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:55:19.542 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:19.542 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:19.543 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:55:19.543 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:19.543 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:55:19.543 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:55:19.543 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:55:19.543 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:19.545 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.533+0000 7f91b804e640 10 monclient: shutdown 2024-03-29T11:55:19.545 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.533+0000 7f91b804e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:55:19.545 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.533+0000 7f91b804e640 1 -- 172.21.15.98:0/2476545554 >> v1:172.21.15.121:6789/0 conn(0x7f91b0152980 legacy=0x7f91b0152df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:19.545 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.533+0000 7f91b804e640 1 -- 172.21.15.98:0/2476545554 shutdown_connections 2024-03-29T11:55:19.545 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.533+0000 7f91b804e640 1 -- 172.21.15.98:0/2476545554 >> 172.21.15.98:0/2476545554 conn(0x7f91b00b35e0 msgr2=0x7f91b00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:55:19.545 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.533+0000 7f91b804e640 1 -- 172.21.15.98:0/2476545554 shutdown_connections 2024-03-29T11:55:19.545 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.533+0000 7f91b804e640 1 -- 172.21.15.98:0/2476545554 wait complete. 2024-03-29T11:55:19.545 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.534+0000 7f91b804e640 10 monclient: build_initial_monmap 2024-03-29T11:55:19.545 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.534+0000 7f91b804e640 10 monclient: monmap: 2024-03-29T11:55:19.545 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:55:19.545 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:55:19.546 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:55:19.546 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:55:19.546 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:55:19.546 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:19.546 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:55:19.546 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:55:19.546 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:55:19.546 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:19.546 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.534+0000 7f91b804e640 1 librados: starting msgr at 2024-03-29T11:55:19.546 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.534+0000 7f91b804e640 1 librados: starting objecter 2024-03-29T11:55:19.546 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.535+0000 7f91b804e640 1 Processor -- start 2024-03-29T11:55:19.546 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.535+0000 7f91b804e640 1 -- start start 2024-03-29T11:55:19.546 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.535+0000 7f91b804e640 1 librados: setting wanted keys 2024-03-29T11:55:19.547 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.535+0000 7f91b804e640 1 librados: calling monclient init 2024-03-29T11:55:19.547 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.535+0000 7f91b804e640 10 monclient: init 2024-03-29T11:55:19.547 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.536+0000 7f91b804e640 10 monclient: _reopen_session rank -1 2024-03-29T11:55:19.547 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.536+0000 7f91b804e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:55:19.547 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.536+0000 7f91b804e640 10 monclient(hunting): picked mon.noname-c con 0x7f91b00ad6c0 addr v1:172.21.15.98:6790/0 2024-03-29T11:55:19.547 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.539+0000 7f91b804e640 10 monclient(hunting): picked mon.noname-a con 0x7f91b0152980 addr v1:172.21.15.98:6789/0 2024-03-29T11:55:19.548 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.539+0000 7f91b55e9640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f91b0152980 0x7f91b015b310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44811/0 (socket says 172.21.15.98:44811) 2024-03-29T11:55:19.548 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.539+0000 7f91b55e9640 1 -- 172.21.15.98:0/4282149180 learned_addr learned my addr 172.21.15.98:0/4282149180 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:55:19.548 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.539+0000 7f91a27fc640 10 client.?.objecter ms_handle_connect 0x7f91b0152980 2024-03-29T11:55:19.548 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.539+0000 7f91a27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:55:19.555 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.539+0000 7f91b804e640 10 monclient(hunting): picked mon.noname-b con 0x7f91b01588d0 addr v1:172.21.15.121:6789/0 2024-03-29T11:55:19.555 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.539+0000 7f91b804e640 1 -- 172.21.15.98:0/4282149180 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91b0154340 con 0x7f91b0152980 2024-03-29T11:55:19.555 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.539+0000 7f91b804e640 1 -- 172.21.15.98:0/4282149180 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91b0154620 con 0x7f91b01588d0 2024-03-29T11:55:19.555 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.539+0000 7f91b804e640 1 -- 172.21.15.98:0/4282149180 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91b015add0 con 0x7f91b00ad6c0 2024-03-29T11:55:19.555 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.539+0000 7f91b804e640 10 monclient(hunting): _renew_subs 2024-03-29T11:55:19.555 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.539+0000 7f91b804e640 10 monclient(hunting): authenticate will time out at 1226.624997s 2024-03-29T11:55:19.555 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.542+0000 7f91a27fc640 1 -- 172.21.15.98:0/4282149180 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1916671037 0 0) 0x7f91b0154340 con 0x7f91b0152980 2024-03-29T11:55:19.555 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.542+0000 7f91a27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:55:19.555 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.542+0000 7f91a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:55:19.555 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.542+0000 7f91a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:55:19.555 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.542+0000 7f91a27fc640 10 monclient(hunting): my global_id is 4464 2024-03-29T11:55:19.555 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.542+0000 7f91a27fc640 1 -- 172.21.15.98:0/4282149180 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9194004eb0 con 0x7f91b0152980 2024-03-29T11:55:19.556 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.543+0000 7f91a27fc640 1 -- 172.21.15.98:0/4282149180 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3681536077 0 0) 0x7f9194004eb0 con 0x7f91b0152980 2024-03-29T11:55:19.556 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.543+0000 7f91a27fc640 1 -- 172.21.15.98:0/4282149180 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91b0154340 con 0x7f91b0152980 2024-03-29T11:55:19.556 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.543+0000 7f91a27fc640 1 -- 172.21.15.98:0/4282149180 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f91a4001610 con 0x7f91b0152980 2024-03-29T11:55:19.556 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.543+0000 7f91a27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:55:19.556 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.543+0000 7f91a27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:55:19.556 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.543+0000 7f91a27fc640 10 monclient(hunting): dump: 2024-03-29T11:55:19.556 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:55:19.556 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:55:19.556 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:19.556 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:19.556 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:55:19.556 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:19.556 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:55:19.556 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:55:19.557 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:55:19.557 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:19.557 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.543+0000 7f91a27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:55:19.557 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.543+0000 7f91a27fc640 1 -- 172.21.15.98:0/4282149180 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 918383624 0 0) 0x7f91b0154340 con 0x7f91b0152980 2024-03-29T11:55:19.557 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.543+0000 7f91a27fc640 1 -- 172.21.15.98:0/4282149180 >> v1:172.21.15.98:6790/0 conn(0x7f91b00ad6c0 legacy=0x7f91b00b2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:19.557 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.543+0000 7f91a27fc640 1 -- 172.21.15.98:0/4282149180 >> v1:172.21.15.121:6789/0 conn(0x7f91b01588d0 legacy=0x7f91b02001d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:19.557 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.543+0000 7f91a27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:55:19.557 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.543+0000 7f91a27fc640 1 monclient: found mon.a 2024-03-29T11:55:19.557 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.543+0000 7f91a27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:55:19.557 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.543+0000 7f91a27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:55:19.557 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.543+0000 7f91a27fc640 1 -- 172.21.15.98:0/4282149180 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b015ba20 con 0x7f91b0152980 2024-03-29T11:55:19.557 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.543+0000 7f91a27fc640 10 monclient: _finish_auth 0 2024-03-29T11:55:19.557 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.543+0000 7f91a27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:55:19.557 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.543+0000 7f91a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:55:19.557 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.543+0000 7f91a27fc640 1 -- 172.21.15.98:0/4282149180 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91a4004bf0 con 0x7f91b0152980 2024-03-29T11:55:19.558 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.543+0000 7f91a27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:55:19.558 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.543+0000 7f91a27fc640 1 -- 172.21.15.98:0/4282149180 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f91a4004f40 con 0x7f91b0152980 2024-03-29T11:55:19.558 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.543+0000 7f91a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:55:19.558 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.543+0000 7f91a27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:55:19.558 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.543+0000 7f91a27fc640 10 monclient: dump: 2024-03-29T11:55:19.558 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:55:19.558 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:55:19.558 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:19.558 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:19.558 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:55:19.558 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:19.558 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:55:19.558 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:55:19.558 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:55:19.559 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:19.559 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.543+0000 7f91a3fff640 4 set_mon_vals no callback set 2024-03-29T11:55:19.559 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.544+0000 7f91b804e640 5 monclient: authenticate success, global_id 4464 2024-03-29T11:55:19.559 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.544+0000 7f91b804e640 10 monclient: _renew_subs 2024-03-29T11:55:19.559 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.544+0000 7f91b804e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:55:19.559 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.544+0000 7f91b804e640 1 -- 172.21.15.98:0/4282149180 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91b00886a0 con 0x7f91b0152980 2024-03-29T11:55:19.559 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.544+0000 7f91b804e640 10 client.4464.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:55:19.559 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.544+0000 7f91b804e640 10 monclient: _renew_subs 2024-03-29T11:55:19.559 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.544+0000 7f91b804e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:55:19.559 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.544+0000 7f91b804e640 1 -- 172.21.15.98:0/4282149180 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f91b0089450 con 0x7f91b0152980 2024-03-29T11:55:19.559 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.544+0000 7f91b804e640 1 librados: init done 2024-03-29T11:55:19.559 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.546+0000 7f91b804e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:55:19.559 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.546+0000 7f91b804e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:55:19.559 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.546+0000 7f91b804e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:55:19.559 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.546+0000 7f91b804e640 1 -- 172.21.15.98:0/4282149180 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91b0075f10 con 0x7f91b0152980 2024-03-29T11:55:19.561 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.550+0000 7f91a27fc640 1 -- 172.21.15.98:0/4282149180 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f91a401af90 con 0x7f91b0152980 2024-03-29T11:55:19.561 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.550+0000 7f91a27fc640 1 -- 172.21.15.98:0/4282149180 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f91a4084cf0 con 0x7f91b0152980 2024-03-29T11:55:19.561 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.550+0000 7f91a27fc640 10 client.4464.objecter ms_dispatch 0x7f91b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:55:19.561 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.550+0000 7f91a27fc640 3 client.4464.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:55:19.561 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.550+0000 7f91a27fc640 3 client.4464.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:55:19.561 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.551+0000 7f91a27fc640 20 client.4464.objecter dump_active .. 0 homeless 2024-03-29T11:55:19.561 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.551+0000 7f91b5dea640 10 monclient: get_auth_request con 0x7f919406dbc0 auth_method 0 2024-03-29T11:55:19.561 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.551+0000 7f91a27fc640 10 client.4464.objecter ms_handle_connect 0x7f919406dbc0 2024-03-29T11:55:19.563 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.555+0000 7f91a27fc640 1 -- 172.21.15.98:0/4282149180 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f91a40505d0 con 0x7f91b0152980 2024-03-29T11:55:19.563 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.555+0000 7f91a27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:55:19.563 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.555+0000 7f91a27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:55:19.890 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.881+0000 7f91b804e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:55:19.890 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.881+0000 7f91b804e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:55:19.890 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.881+0000 7f91b804e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:55:19.890 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.881+0000 7f91b804e640 1 -- 172.21.15.98:0/4282149180 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f91b0088830 con 0x7f91b0152980 2024-03-29T11:55:19.891 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.882+0000 7f91a27fc640 1 -- 172.21.15.98:0/4282149180 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f91a4052500 con 0x7f91b0152980 2024-03-29T11:55:19.891 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.882+0000 7f91a27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:55:19.891 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.882+0000 7f91a27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:55:19.891 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:55:19.891 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:55:19.895 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.886+0000 7f91b804e640 10 librados: watch_flush enter 2024-03-29T11:55:19.895 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.886+0000 7f91b804e640 10 librados: watch_flush exit 2024-03-29T11:55:19.895 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.886+0000 7f91b804e640 20 client.4464.objecter shutdown clearing up homeless session... 2024-03-29T11:55:19.895 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.886+0000 7f91b804e640 10 client.4464.objecter successfully canceled tick 2024-03-29T11:55:19.895 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.886+0000 7f91b804e640 1 -- 172.21.15.98:0/4282149180 >> v1:172.21.15.98:6816/4065186974 conn(0x7f919406dbc0 legacy=0x7f9194070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:19.895 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.886+0000 7f91b804e640 10 monclient: shutdown 2024-03-29T11:55:19.895 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.886+0000 7f91b804e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:55:19.895 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.886+0000 7f91b804e640 1 -- 172.21.15.98:0/4282149180 >> v1:172.21.15.98:6789/0 conn(0x7f91b0152980 legacy=0x7f91b015b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:19.895 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.886+0000 7f91b804e640 1 -- 172.21.15.98:0/4282149180 shutdown_connections 2024-03-29T11:55:19.895 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.886+0000 7f91b804e640 1 -- 172.21.15.98:0/4282149180 >> 172.21.15.98:0/4282149180 conn(0x7f91b00b35e0 msgr2=0x7f91b014f8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:55:19.895 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.887+0000 7f91b804e640 1 -- 172.21.15.98:0/4282149180 shutdown_connections 2024-03-29T11:55:19.895 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.887+0000 7f91b804e640 1 -- 172.21.15.98:0/4282149180 wait complete. 2024-03-29T11:55:19.895 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:19.887+0000 7f91b804e640 1 librados: shutdown 2024-03-29T11:55:19.911 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:55:26.718 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:55:26.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.861+0000 7f20dffff640 10 monclient: get_monmap_and_config 2024-03-29T11:55:26.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.861+0000 7f20dffff640 10 monclient: build_initial_monmap 2024-03-29T11:55:26.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.862+0000 7f20dffff640 10 monclient: monmap: 2024-03-29T11:55:26.870 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:55:26.870 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:55:26.871 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:55:26.862886+0000 2024-03-29T11:55:26.871 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:55:26.862886+0000 2024-03-29T11:55:26.871 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:55:26.871 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:26.871 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:55:26.871 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:55:26.871 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:55:26.871 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:26.871 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.863+0000 7f20dffff640 1 Processor -- start 2024-03-29T11:55:26.879 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.865+0000 7f20dffff640 1 -- start start 2024-03-29T11:55:26.879 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.865+0000 7f20dffff640 10 monclient: init 2024-03-29T11:55:26.879 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.866+0000 7f20dffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:55:26.879 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.866+0000 7f20dffff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:55:26.879 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.866+0000 7f20dffff640 10 monclient(hunting): picked mon.noname-c con 0x7f20e00a9750 addr v1:172.21.15.121:6789/0 2024-03-29T11:55:26.879 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.866+0000 7f20dffff640 10 monclient(hunting): picked mon.noname-b con 0x7f20e00add40 addr v1:172.21.15.98:6790/0 2024-03-29T11:55:26.879 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.866+0000 7f20dffff640 10 monclient(hunting): picked mon.noname-a con 0x7f20e00aad20 addr v1:172.21.15.98:6789/0 2024-03-29T11:55:26.879 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.866+0000 7f20dffff640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20e0149870 con 0x7f20e00aad20 2024-03-29T11:55:26.879 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.866+0000 7f20dffff640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20e00abf80 con 0x7f20e00a9750 2024-03-29T11:55:26.879 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.866+0000 7f20dffff640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20e0149ac0 con 0x7f20e00add40 2024-03-29T11:55:26.879 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.866+0000 7f20dffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:55:26.880 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.866+0000 7f20dffff640 10 monclient(hunting): authenticate will time out at 1233.951633s 2024-03-29T11:55:26.880 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.868+0000 7f20de7fc640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f20e00add40 0x7f20e0143610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:40080/0 (socket says 172.21.15.98:40080) 2024-03-29T11:55:26.880 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.868+0000 7f20de7fc640 1 -- 172.21.15.98:0/3021813898 learned_addr learned my addr 172.21.15.98:0/3021813898 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:55:26.880 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.869+0000 7f20ddffb640 1 -- 172.21.15.98:0/3021813898 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3115303543 0 0) 0x7f20e0149870 con 0x7f20e00aad20 2024-03-29T11:55:26.880 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.870+0000 7f20ddffb640 10 monclient(hunting): _negotiate 2024-03-29T11:55:26.880 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.870+0000 7f20ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:55:26.880 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.870+0000 7f20ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:55:26.880 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.870+0000 7f20ddffb640 10 monclient(hunting): my global_id is 4467 2024-03-29T11:55:26.880 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.870+0000 7f20ddffb640 1 -- 172.21.15.98:0/3021813898 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20c8003520 con 0x7f20e00aad20 2024-03-29T11:55:26.880 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.870+0000 7f20ddffb640 1 -- 172.21.15.98:0/3021813898 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3208220274 0 0) 0x7f20e0149ac0 con 0x7f20e00add40 2024-03-29T11:55:26.881 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.870+0000 7f20ddffb640 10 monclient(hunting): _negotiate 2024-03-29T11:55:26.881 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.870+0000 7f20ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:55:26.881 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.870+0000 7f20ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:55:26.881 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.870+0000 7f20ddffb640 10 monclient(hunting): my global_id is 4502 2024-03-29T11:55:26.881 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.870+0000 7f20ddffb640 1 -- 172.21.15.98:0/3021813898 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20e0149870 con 0x7f20e00add40 2024-03-29T11:55:26.881 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.870+0000 7f20ddffb640 1 -- 172.21.15.98:0/3021813898 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 251368186 0 0) 0x7f20c8003520 con 0x7f20e00aad20 2024-03-29T11:55:26.881 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.870+0000 7f20ddffb640 1 -- 172.21.15.98:0/3021813898 >> v1:172.21.15.98:6790/0 conn(0x7f20e00add40 legacy=0x7f20e0143610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:26.881 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.870+0000 7f20ddffb640 1 -- 172.21.15.98:0/3021813898 >> v1:172.21.15.121:6789/0 conn(0x7f20e00a9750 legacy=0x7f20e0149fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:55:26.881 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.870+0000 7f20ddffb640 10 monclient: _finish_hunting 0 2024-03-29T11:55:26.881 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.870+0000 7f20ddffb640 1 monclient: found mon.noname-a 2024-03-29T11:55:26.881 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.870+0000 7f20ddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:55:26.881 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.870+0000 7f20ddffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T11:55:26.881 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.870+0000 7f20ddffb640 1 -- 172.21.15.98:0/3021813898 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e014a6f0 con 0x7f20e00aad20 2024-03-29T11:55:26.881 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.870+0000 7f20ddffb640 10 monclient: _finish_auth 0 2024-03-29T11:55:26.881 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.870+0000 7f20ddffb640 10 monclient: _check_auth_tickets 2024-03-29T11:55:26.882 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.870+0000 7f20ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:55:26.882 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.870+0000 7f20ddffb640 1 -- 172.21.15.98:0/3021813898 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f20d8001a10 con 0x7f20e00aad20 2024-03-29T11:55:26.882 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.870+0000 7f20ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:55:26.882 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.870+0000 7f20ddffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:55:26.882 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.870+0000 7f20ddffb640 10 monclient: dump: 2024-03-29T11:55:26.882 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:55:26.882 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:55:26.882 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:26.882 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:26.882 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:55:26.882 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:26.882 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:55:26.882 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:55:26.882 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:55:26.883 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:26.883 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.870+0000 7f20dffff640 5 monclient: authenticate success, global_id 4467 2024-03-29T11:55:26.883 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.870+0000 7f20dffff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:55:26.883 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.871+0000 7f20ddffb640 1 -- 172.21.15.98:0/3021813898 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20d8001bf0 con 0x7f20e00aad20 2024-03-29T11:55:26.883 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.871+0000 7f20ddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:55:26.883 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.871+0000 7f20dffff640 10 monclient: get_monmap_and_config success 2024-03-29T11:55:26.883 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.871+0000 7f20dffff640 4 set_mon_vals no callback set 2024-03-29T11:55:26.886 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.871+0000 7f20dffff640 10 monclient: shutdown 2024-03-29T11:55:26.886 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.871+0000 7f20dffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:55:26.886 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.871+0000 7f20dffff640 1 -- 172.21.15.98:0/3021813898 >> v1:172.21.15.98:6789/0 conn(0x7f20e00aad20 legacy=0x7f20e0151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:26.886 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.872+0000 7f20dffff640 1 -- 172.21.15.98:0/3021813898 shutdown_connections 2024-03-29T11:55:26.886 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.872+0000 7f20dffff640 1 -- 172.21.15.98:0/3021813898 >> 172.21.15.98:0/3021813898 conn(0x7f20e00673e0 msgr2=0x7f20e00b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:55:26.886 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.872+0000 7f20dffff640 1 -- 172.21.15.98:0/3021813898 shutdown_connections 2024-03-29T11:55:26.886 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.872+0000 7f20dffff640 1 -- 172.21.15.98:0/3021813898 wait complete. 2024-03-29T11:55:26.886 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.873+0000 7f20dffff640 10 monclient: build_initial_monmap 2024-03-29T11:55:26.886 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.873+0000 7f20dffff640 10 monclient: monmap: 2024-03-29T11:55:26.887 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:55:26.887 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:55:26.887 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:55:26.887 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:55:26.887 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:55:26.887 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:26.887 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:55:26.887 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:55:26.887 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:55:26.887 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:26.887 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.873+0000 7f20dffff640 1 librados: starting msgr at 2024-03-29T11:55:26.887 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.873+0000 7f20dffff640 1 librados: starting objecter 2024-03-29T11:55:26.888 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.874+0000 7f20dffff640 1 Processor -- start 2024-03-29T11:55:26.888 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.874+0000 7f20dffff640 1 -- start start 2024-03-29T11:55:26.888 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.874+0000 7f20dffff640 1 librados: setting wanted keys 2024-03-29T11:55:26.888 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.874+0000 7f20dffff640 1 librados: calling monclient init 2024-03-29T11:55:26.888 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.874+0000 7f20dffff640 10 monclient: init 2024-03-29T11:55:26.888 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.875+0000 7f20dffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:55:26.888 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.875+0000 7f20dffff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:55:26.888 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.875+0000 7f20dffff640 10 monclient(hunting): picked mon.noname-a con 0x7f20e00a9750 addr v1:172.21.15.98:6789/0 2024-03-29T11:55:26.888 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.875+0000 7f20dffff640 10 monclient(hunting): picked mon.noname-b con 0x7f20e00aad20 addr v1:172.21.15.121:6789/0 2024-03-29T11:55:26.888 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.875+0000 7f20dffff640 10 monclient(hunting): picked mon.noname-c con 0x7f20e00add40 addr v1:172.21.15.98:6790/0 2024-03-29T11:55:26.888 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.875+0000 7f20dffff640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20e013ef50 con 0x7f20e00a9750 2024-03-29T11:55:26.889 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.875+0000 7f20dffff640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20e006f410 con 0x7f20e00aad20 2024-03-29T11:55:26.889 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.875+0000 7f20dffff640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20e0070170 con 0x7f20e00add40 2024-03-29T11:55:26.889 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.875+0000 7f20dffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:55:26.889 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.875+0000 7f20dffff640 10 monclient(hunting): authenticate will time out at 1233.960750s 2024-03-29T11:55:26.891 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.878+0000 7f20df7fe640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f20e00add40 0x7f20e0071bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:42536/0 (socket says 172.21.15.98:42536) 2024-03-29T11:55:26.891 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.878+0000 7f20df7fe640 1 -- 172.21.15.98:0/930386056 learned_addr learned my addr 172.21.15.98:0/930386056 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:55:26.891 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.879+0000 7f20c77fe640 10 client.?.objecter ms_handle_connect 0x7f20e00aad20 2024-03-29T11:55:26.891 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.879+0000 7f20c77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:55:26.891 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.880+0000 7f20c77fe640 1 -- 172.21.15.98:0/930386056 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2770695871 0 0) 0x7f20e006f410 con 0x7f20e00aad20 2024-03-29T11:55:26.891 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.880+0000 7f20c77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:55:26.891 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.880+0000 7f20c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:55:26.891 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.880+0000 7f20c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:55:26.891 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.880+0000 7f20c77fe640 10 monclient(hunting): my global_id is 4486 2024-03-29T11:55:26.891 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.880+0000 7f20c77fe640 1 -- 172.21.15.98:0/930386056 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20cc004eb0 con 0x7f20e00aad20 2024-03-29T11:55:26.892 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.881+0000 7f20c77fe640 1 -- 172.21.15.98:0/930386056 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1925926243 0 0) 0x7f20cc004eb0 con 0x7f20e00aad20 2024-03-29T11:55:26.892 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.881+0000 7f20c77fe640 1 -- 172.21.15.98:0/930386056 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20e006f410 con 0x7f20e00aad20 2024-03-29T11:55:26.892 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.881+0000 7f20c77fe640 1 -- 172.21.15.98:0/930386056 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f20d4001850 con 0x7f20e00aad20 2024-03-29T11:55:26.892 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.881+0000 7f20c77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:55:26.892 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.881+0000 7f20c77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:55:26.892 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.881+0000 7f20c77fe640 10 monclient(hunting): dump: 2024-03-29T11:55:26.892 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:55:26.892 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:55:26.892 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:26.892 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:26.892 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:55:26.892 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:26.892 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:55:26.892 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:55:26.892 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:55:26.893 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:26.893 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.881+0000 7f20c77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:55:26.893 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.881+0000 7f20c77fe640 10 client.?.objecter ms_handle_connect 0x7f20e00add40 2024-03-29T11:55:26.893 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.881+0000 7f20c77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:55:26.893 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.882+0000 7f20c77fe640 10 client.?.objecter ms_handle_connect 0x7f20e00a9750 2024-03-29T11:55:26.893 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.882+0000 7f20c77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:55:26.893 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.882+0000 7f20c77fe640 1 -- 172.21.15.98:0/930386056 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1065634344 0 0) 0x7f20e006f410 con 0x7f20e00aad20 2024-03-29T11:55:26.893 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.882+0000 7f20c77fe640 1 -- 172.21.15.98:0/930386056 >> v1:172.21.15.98:6790/0 conn(0x7f20e00add40 legacy=0x7f20e0071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:26.893 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.882+0000 7f20c77fe640 1 -- 172.21.15.98:0/930386056 >> v1:172.21.15.98:6789/0 conn(0x7f20e00a9750 legacy=0x7f20e006ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:26.893 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.882+0000 7f20c77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:55:26.893 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.882+0000 7f20c77fe640 1 monclient: found mon.b 2024-03-29T11:55:26.893 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.882+0000 7f20c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:55:26.893 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.882+0000 7f20c77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:55:26.893 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.883+0000 7f20c77fe640 1 -- 172.21.15.98:0/930386056 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e0070450 con 0x7f20e00aad20 2024-03-29T11:55:26.894 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.883+0000 7f20c77fe640 10 monclient: _finish_auth 0 2024-03-29T11:55:26.894 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.883+0000 7f20c77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:55:26.894 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.883+0000 7f20c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:55:26.895 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.884+0000 7f20dffff640 5 monclient: authenticate success, global_id 4486 2024-03-29T11:55:26.895 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.884+0000 7f20dffff640 10 monclient: _renew_subs 2024-03-29T11:55:26.896 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.884+0000 7f20dffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:55:26.896 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.884+0000 7f20dffff640 1 -- 172.21.15.98:0/930386056 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20e006e060 con 0x7f20e00aad20 2024-03-29T11:55:26.896 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.884+0000 7f20dffff640 10 client.4486.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:55:26.896 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.884+0000 7f20dffff640 10 monclient: _renew_subs 2024-03-29T11:55:26.896 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.884+0000 7f20dffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:55:26.896 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.884+0000 7f20dffff640 1 -- 172.21.15.98:0/930386056 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f20e01dbe30 con 0x7f20e00aad20 2024-03-29T11:55:26.896 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.884+0000 7f20dffff640 1 librados: init done 2024-03-29T11:55:26.896 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.884+0000 7f20c77fe640 1 -- 172.21.15.98:0/930386056 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20d40019b0 con 0x7f20e00aad20 2024-03-29T11:55:26.896 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.884+0000 7f20c77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:55:26.896 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.884+0000 7f20c77fe640 1 -- 172.21.15.98:0/930386056 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f20d4004a40 con 0x7f20e00aad20 2024-03-29T11:55:26.897 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.884+0000 7f20c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:55:26.897 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.884+0000 7f20c77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:55:26.897 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.884+0000 7f20c77fe640 10 monclient: dump: 2024-03-29T11:55:26.897 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:55:26.897 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:55:26.897 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:26.897 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:26.897 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:55:26.897 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:26.897 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:55:26.897 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:55:26.898 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:55:26.898 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:26.898 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.886+0000 7f20dcff9640 4 set_mon_vals no callback set 2024-03-29T11:55:26.898 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.887+0000 7f20dffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:55:26.898 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.887+0000 7f20dffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:55:26.898 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.887+0000 7f20dffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:55:26.898 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.887+0000 7f20dffff640 1 -- 172.21.15.98:0/930386056 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20c0005080 con 0x7f20e00aad20 2024-03-29T11:55:26.898 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.887+0000 7f20c77fe640 1 -- 172.21.15.98:0/930386056 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f20d401aa90 con 0x7f20e00aad20 2024-03-29T11:55:26.899 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.888+0000 7f20deffd640 10 monclient: get_auth_request con 0x7f20cc06dc30 auth_method 0 2024-03-29T11:55:26.899 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.888+0000 7f20c77fe640 1 -- 172.21.15.98:0/930386056 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f20d4004860 con 0x7f20e00aad20 2024-03-29T11:55:26.899 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.889+0000 7f20c77fe640 10 client.4486.objecter ms_dispatch 0x7f20e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:55:26.900 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.889+0000 7f20c77fe640 3 client.4486.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:55:26.900 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.889+0000 7f20c77fe640 3 client.4486.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:55:26.900 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.889+0000 7f20c77fe640 20 client.4486.objecter dump_active .. 0 homeless 2024-03-29T11:55:26.900 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.889+0000 7f20c77fe640 10 client.4486.objecter ms_handle_connect 0x7f20cc06dc30 2024-03-29T11:55:26.905 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.896+0000 7f20c77fe640 1 -- 172.21.15.98:0/930386056 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f20d4050970 con 0x7f20e00aad20 2024-03-29T11:55:26.905 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.897+0000 7f20c77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:55:26.905 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:26.897+0000 7f20c77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:55:27.231 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:27.222+0000 7f20dffff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:55:27.231 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:27.222+0000 7f20dffff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:55:27.231 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:27.222+0000 7f20dffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:55:27.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:27.222+0000 7f20dffff640 1 -- 172.21.15.98:0/930386056 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20c0005330 con 0x7f20e00aad20 2024-03-29T11:55:27.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:27.223+0000 7f20c77fe640 1 -- 172.21.15.98:0/930386056 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f20d4050970 con 0x7f20e00aad20 2024-03-29T11:55:27.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:27.223+0000 7f20c77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:55:27.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:27.223+0000 7f20c77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:55:27.232 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:55:27.232 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:55:27.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:27.228+0000 7f20dffff640 10 librados: watch_flush enter 2024-03-29T11:55:27.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:27.228+0000 7f20dffff640 10 librados: watch_flush exit 2024-03-29T11:55:27.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:27.228+0000 7f20dffff640 20 client.4486.objecter shutdown clearing up homeless session... 2024-03-29T11:55:27.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:27.228+0000 7f20dffff640 10 client.4486.objecter successfully canceled tick 2024-03-29T11:55:27.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:27.228+0000 7f20dffff640 1 -- 172.21.15.98:0/930386056 >> v1:172.21.15.98:6816/4065186974 conn(0x7f20cc06dc30 legacy=0x7f20cc0700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:27.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:27.228+0000 7f20dffff640 10 monclient: shutdown 2024-03-29T11:55:27.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:27.228+0000 7f20dffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:55:27.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:27.228+0000 7f20dffff640 1 -- 172.21.15.98:0/930386056 >> v1:172.21.15.121:6789/0 conn(0x7f20e00aad20 legacy=0x7f20e01db720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:27.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:27.229+0000 7f20dffff640 1 -- 172.21.15.98:0/930386056 shutdown_connections 2024-03-29T11:55:27.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:27.229+0000 7f20dffff640 1 -- 172.21.15.98:0/930386056 >> 172.21.15.98:0/930386056 conn(0x7f20e00673e0 msgr2=0x7f20e00a9fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:55:27.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:27.229+0000 7f20dffff640 1 -- 172.21.15.98:0/930386056 shutdown_connections 2024-03-29T11:55:27.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:27.229+0000 7f20dffff640 1 -- 172.21.15.98:0/930386056 wait complete. 2024-03-29T11:55:27.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:27.229+0000 7f20dffff640 1 librados: shutdown 2024-03-29T11:55:27.254 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:55:34.060 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:55:34.203 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.194+0000 7fb1b501e640 10 monclient: get_monmap_and_config 2024-03-29T11:55:34.203 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.194+0000 7fb1b501e640 10 monclient: build_initial_monmap 2024-03-29T11:55:34.203 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.194+0000 7fb1b501e640 10 monclient: monmap: 2024-03-29T11:55:34.203 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:55:34.203 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:55:34.203 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:55:34.196053+0000 2024-03-29T11:55:34.203 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:55:34.196053+0000 2024-03-29T11:55:34.204 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:55:34.204 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:34.204 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:55:34.204 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:55:34.204 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:55:34.204 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:34.204 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.195+0000 7fb1b501e640 1 Processor -- start 2024-03-29T11:55:34.204 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.195+0000 7fb1b501e640 1 -- start start 2024-03-29T11:55:34.204 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.195+0000 7fb1b501e640 10 monclient: init 2024-03-29T11:55:34.206 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.196+0000 7fb1b501e640 10 monclient: _reopen_session rank -1 2024-03-29T11:55:34.206 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.196+0000 7fb1b501e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:55:34.206 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.196+0000 7fb1b501e640 10 monclient(hunting): picked mon.noname-b con 0x7fb1a8006890 addr v1:172.21.15.98:6790/0 2024-03-29T11:55:34.207 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.196+0000 7fb1b501e640 10 monclient(hunting): picked mon.noname-a con 0x7fb1a8006c70 addr v1:172.21.15.98:6789/0 2024-03-29T11:55:34.207 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.196+0000 7fb1b501e640 10 monclient(hunting): picked mon.noname-c con 0x7fb1a802b4c0 addr v1:172.21.15.121:6789/0 2024-03-29T11:55:34.207 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.196+0000 7fb1b501e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1a802c830 con 0x7fb1a8006c70 2024-03-29T11:55:34.207 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.196+0000 7fb1b501e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1a802afa0 con 0x7fb1a802b4c0 2024-03-29T11:55:34.207 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.196+0000 7fb1b501e640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1a802aba0 con 0x7fb1a8006890 2024-03-29T11:55:34.207 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.196+0000 7fb1b501e640 10 monclient(hunting): _renew_subs 2024-03-29T11:55:34.207 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.196+0000 7fb1b501e640 10 monclient(hunting): authenticate will time out at 1241.282464s 2024-03-29T11:55:34.207 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.198+0000 7fb1aed9d640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fb1a8006890 0x7fb1a80c0a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:44780/0 (socket says 172.21.15.98:44780) 2024-03-29T11:55:34.207 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.198+0000 7fb1aed9d640 1 -- 172.21.15.98:0/311085964 learned_addr learned my addr 172.21.15.98:0/311085964 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:55:34.208 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.199+0000 7fb1add9b640 1 -- 172.21.15.98:0/311085964 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2738248117 0 0) 0x7fb1a802aba0 con 0x7fb1a8006890 2024-03-29T11:55:34.208 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.199+0000 7fb1add9b640 10 monclient(hunting): _negotiate 2024-03-29T11:55:34.208 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.199+0000 7fb1add9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:55:34.208 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.199+0000 7fb1add9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:55:34.208 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.199+0000 7fb1add9b640 10 monclient(hunting): my global_id is 4505 2024-03-29T11:55:34.208 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.199+0000 7fb1add9b640 1 -- 172.21.15.98:0/311085964 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb198003520 con 0x7fb1a8006890 2024-03-29T11:55:34.209 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.199+0000 7fb1add9b640 1 -- 172.21.15.98:0/311085964 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2630484629 0 0) 0x7fb198003520 con 0x7fb1a8006890 2024-03-29T11:55:34.209 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.199+0000 7fb1add9b640 1 -- 172.21.15.98:0/311085964 >> v1:172.21.15.121:6789/0 conn(0x7fb1a802b4c0 legacy=0x7fb1a80cf0e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:55:34.209 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.199+0000 7fb1add9b640 1 -- 172.21.15.98:0/311085964 >> v1:172.21.15.98:6789/0 conn(0x7fb1a8006c70 legacy=0x7fb1a80c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:55:34.209 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.199+0000 7fb1add9b640 10 monclient: _finish_hunting 0 2024-03-29T11:55:34.209 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.199+0000 7fb1add9b640 1 monclient: found mon.noname-b 2024-03-29T11:55:34.209 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.199+0000 7fb1add9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:55:34.209 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.199+0000 7fb1add9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:55:34.209 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.199+0000 7fb1add9b640 1 -- 172.21.15.98:0/311085964 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1a802ad30 con 0x7fb1a8006890 2024-03-29T11:55:34.209 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.199+0000 7fb1add9b640 10 monclient: _finish_auth 0 2024-03-29T11:55:34.209 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.199+0000 7fb1add9b640 10 monclient: _check_auth_tickets 2024-03-29T11:55:34.210 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.199+0000 7fb1add9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:55:34.210 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.199+0000 7fb1b501e640 5 monclient: authenticate success, global_id 4505 2024-03-29T11:55:34.210 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.199+0000 7fb1b501e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:55:34.210 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.199+0000 7fb1add9b640 1 -- 172.21.15.98:0/311085964 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fb1a4001690 con 0x7fb1a8006890 2024-03-29T11:55:34.210 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.199+0000 7fb1add9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:55:34.210 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.199+0000 7fb1add9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:55:34.210 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.199+0000 7fb1add9b640 10 monclient: dump: 2024-03-29T11:55:34.210 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:55:34.210 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:55:34.210 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:34.210 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:34.210 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:55:34.210 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:34.210 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:55:34.211 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:55:34.211 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:55:34.211 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:34.211 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.199+0000 7fb1b501e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:55:34.211 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.199+0000 7fb1add9b640 1 -- 172.21.15.98:0/311085964 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1a40018b0 con 0x7fb1a8006890 2024-03-29T11:55:34.211 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.199+0000 7fb1add9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:55:34.211 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.199+0000 7fb1b501e640 10 monclient: get_monmap_and_config success 2024-03-29T11:55:34.211 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.199+0000 7fb1b501e640 4 set_mon_vals no callback set 2024-03-29T11:55:34.212 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.200+0000 7fb1add9b640 1 -- 172.21.15.98:0/311085964 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fb1a4001c40 con 0x7fb1a8006890 2024-03-29T11:55:34.212 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.200+0000 7fb1add9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:55:34.212 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.200+0000 7fb1add9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:55:34.212 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.200+0000 7fb1add9b640 10 monclient: dump: 2024-03-29T11:55:34.212 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:55:34.212 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:55:34.212 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:34.212 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:34.213 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:55:34.213 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:34.213 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:55:34.213 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:55:34.213 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:55:34.213 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:34.213 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.202+0000 7fb1b501e640 10 monclient: shutdown 2024-03-29T11:55:34.213 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.202+0000 7fb1b501e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:55:34.213 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.202+0000 7fb1b501e640 1 -- 172.21.15.98:0/311085964 >> v1:172.21.15.98:6790/0 conn(0x7fb1a8006890 legacy=0x7fb1a80c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:34.213 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.202+0000 7fb1b501e640 1 -- 172.21.15.98:0/311085964 shutdown_connections 2024-03-29T11:55:34.213 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.202+0000 7fb1b501e640 1 -- 172.21.15.98:0/311085964 >> 172.21.15.98:0/311085964 conn(0x7fb1a80058f0 msgr2=0x7fb1a8005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:55:34.213 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.203+0000 7fb1b501e640 1 -- 172.21.15.98:0/311085964 shutdown_connections 2024-03-29T11:55:34.213 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.203+0000 7fb1b501e640 1 -- 172.21.15.98:0/311085964 wait complete. 2024-03-29T11:55:34.214 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.204+0000 7fb1b501e640 10 monclient: build_initial_monmap 2024-03-29T11:55:34.214 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.204+0000 7fb1b501e640 10 monclient: monmap: 2024-03-29T11:55:34.214 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:55:34.214 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:55:34.214 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:55:34.214 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:55:34.214 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:55:34.214 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:34.214 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:55:34.215 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:55:34.215 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:55:34.215 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:34.216 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.204+0000 7fb1b501e640 1 librados: starting msgr at 2024-03-29T11:55:34.216 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.204+0000 7fb1b501e640 1 librados: starting objecter 2024-03-29T11:55:34.216 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.205+0000 7fb1b501e640 1 Processor -- start 2024-03-29T11:55:34.216 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.205+0000 7fb1b501e640 1 -- start start 2024-03-29T11:55:34.216 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.205+0000 7fb1b501e640 1 librados: setting wanted keys 2024-03-29T11:55:34.216 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.205+0000 7fb1b501e640 1 librados: calling monclient init 2024-03-29T11:55:34.217 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.205+0000 7fb1b501e640 10 monclient: init 2024-03-29T11:55:34.217 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.205+0000 7fb1b501e640 10 monclient: _reopen_session rank -1 2024-03-29T11:55:34.217 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.205+0000 7fb1b501e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:55:34.217 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.205+0000 7fb1b501e640 10 monclient(hunting): picked mon.noname-a con 0x7fb1a8006890 addr v1:172.21.15.98:6789/0 2024-03-29T11:55:34.217 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.205+0000 7fb1b501e640 10 monclient(hunting): picked mon.noname-c con 0x7fb1a802b4c0 addr v1:172.21.15.98:6790/0 2024-03-29T11:55:34.217 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.205+0000 7fb1b501e640 10 monclient(hunting): picked mon.noname-b con 0x7fb1a80c53f0 addr v1:172.21.15.121:6789/0 2024-03-29T11:55:34.217 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.205+0000 7fb1b501e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1a80c4740 con 0x7fb1a8006890 2024-03-29T11:55:34.217 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.205+0000 7fb1b501e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1a80c34b0 con 0x7fb1a80c53f0 2024-03-29T11:55:34.217 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.205+0000 7fb1b501e640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1a8160ea0 con 0x7fb1a802b4c0 2024-03-29T11:55:34.217 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.205+0000 7fb1b501e640 10 monclient(hunting): _renew_subs 2024-03-29T11:55:34.217 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.205+0000 7fb1b501e640 10 monclient(hunting): authenticate will time out at 1241.292006s 2024-03-29T11:55:34.218 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.209+0000 7fb1af59e640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fb1a80c53f0 0x7fb1a80bb8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:59777/0 (socket says 172.21.15.98:59777) 2024-03-29T11:55:34.220 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.209+0000 7fb1af59e640 1 -- 172.21.15.98:0/2701802024 learned_addr learned my addr 172.21.15.98:0/2701802024 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:55:34.220 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.210+0000 7fb1977fe640 10 client.?.objecter ms_handle_connect 0x7fb1a80c53f0 2024-03-29T11:55:34.220 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.210+0000 7fb1977fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:55:34.220 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.210+0000 7fb1977fe640 10 client.?.objecter ms_handle_connect 0x7fb1a8006890 2024-03-29T11:55:34.220 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.210+0000 7fb1977fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:55:34.220 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.210+0000 7fb1977fe640 1 -- 172.21.15.98:0/2701802024 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2858098849 0 0) 0x7fb1a80c4740 con 0x7fb1a8006890 2024-03-29T11:55:34.220 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.210+0000 7fb1977fe640 10 monclient(hunting): _negotiate 2024-03-29T11:55:34.220 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.210+0000 7fb1977fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:55:34.221 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.210+0000 7fb1977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:55:34.221 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.210+0000 7fb1977fe640 10 monclient(hunting): my global_id is 4473 2024-03-29T11:55:34.221 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.210+0000 7fb1977fe640 1 -- 172.21.15.98:0/2701802024 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb19c004eb0 con 0x7fb1a8006890 2024-03-29T11:55:34.221 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.211+0000 7fb1977fe640 10 client.?.objecter ms_handle_connect 0x7fb1a802b4c0 2024-03-29T11:55:34.221 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.211+0000 7fb1977fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:55:34.223 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.211+0000 7fb1977fe640 1 -- 172.21.15.98:0/2701802024 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 978137750 0 0) 0x7fb19c004eb0 con 0x7fb1a8006890 2024-03-29T11:55:34.224 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.211+0000 7fb1977fe640 1 -- 172.21.15.98:0/2701802024 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1a80c4740 con 0x7fb1a8006890 2024-03-29T11:55:34.224 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.211+0000 7fb1977fe640 1 -- 172.21.15.98:0/2701802024 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fb1a4001550 con 0x7fb1a8006890 2024-03-29T11:55:34.224 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.211+0000 7fb1977fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:55:34.224 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.211+0000 7fb1977fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:55:34.224 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.211+0000 7fb1977fe640 10 monclient(hunting): dump: 2024-03-29T11:55:34.224 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:55:34.224 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:55:34.224 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:34.224 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:34.224 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:55:34.224 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:34.224 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:55:34.224 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:55:34.225 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:55:34.225 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:34.225 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.211+0000 7fb1977fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:55:34.225 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.211+0000 7fb1977fe640 1 -- 172.21.15.98:0/2701802024 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 999732098 0 0) 0x7fb1a80c4740 con 0x7fb1a8006890 2024-03-29T11:55:34.225 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.211+0000 7fb1977fe640 1 -- 172.21.15.98:0/2701802024 >> v1:172.21.15.98:6790/0 conn(0x7fb1a802b4c0 legacy=0x7fb1a81613c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:34.225 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.211+0000 7fb1977fe640 1 -- 172.21.15.98:0/2701802024 >> v1:172.21.15.121:6789/0 conn(0x7fb1a80c53f0 legacy=0x7fb1a80bb8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:34.225 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.211+0000 7fb1977fe640 10 monclient: _finish_hunting 0 2024-03-29T11:55:34.225 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.211+0000 7fb1977fe640 1 monclient: found mon.a 2024-03-29T11:55:34.225 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.211+0000 7fb1977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:55:34.225 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.211+0000 7fb1977fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:55:34.225 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.211+0000 7fb1977fe640 1 -- 172.21.15.98:0/2701802024 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1a8161ad0 con 0x7fb1a8006890 2024-03-29T11:55:34.225 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.211+0000 7fb1977fe640 10 monclient: _finish_auth 0 2024-03-29T11:55:34.225 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.211+0000 7fb1977fe640 10 monclient: _check_auth_tickets 2024-03-29T11:55:34.225 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.211+0000 7fb1977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:55:34.226 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.212+0000 7fb1977fe640 1 -- 172.21.15.98:0/2701802024 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1a4004a90 con 0x7fb1a8006890 2024-03-29T11:55:34.226 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.212+0000 7fb1977fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:55:34.226 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.212+0000 7fb1977fe640 1 -- 172.21.15.98:0/2701802024 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fb1a4004da0 con 0x7fb1a8006890 2024-03-29T11:55:34.226 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.212+0000 7fb1977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:55:34.226 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.212+0000 7fb1977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:55:34.226 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.212+0000 7fb1977fe640 10 monclient: dump: 2024-03-29T11:55:34.226 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:55:34.226 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:55:34.226 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:34.226 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:34.226 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:55:34.226 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:34.226 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:55:34.227 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:55:34.227 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:55:34.227 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:34.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.212+0000 7fb1acd99640 4 set_mon_vals no callback set 2024-03-29T11:55:34.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.213+0000 7fb1b501e640 5 monclient: authenticate success, global_id 4473 2024-03-29T11:55:34.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.213+0000 7fb1b501e640 10 monclient: _renew_subs 2024-03-29T11:55:34.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.213+0000 7fb1b501e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:55:34.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.213+0000 7fb1b501e640 1 -- 172.21.15.98:0/2701802024 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1a80bbfc0 con 0x7fb1a8006890 2024-03-29T11:55:34.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.213+0000 7fb1b501e640 10 client.4473.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:55:34.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.213+0000 7fb1b501e640 10 monclient: _renew_subs 2024-03-29T11:55:34.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.213+0000 7fb1b501e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:55:34.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.213+0000 7fb1b501e640 1 -- 172.21.15.98:0/2701802024 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1a80c24c0 con 0x7fb1a8006890 2024-03-29T11:55:34.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.213+0000 7fb1b501e640 1 librados: init done 2024-03-29T11:55:34.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.215+0000 7fb1b501e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:55:34.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.215+0000 7fb1b501e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:55:34.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.215+0000 7fb1b501e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:55:34.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.215+0000 7fb1b501e640 1 -- 172.21.15.98:0/2701802024 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1a8160d00 con 0x7fb1a8006890 2024-03-29T11:55:34.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.217+0000 7fb1977fe640 1 -- 172.21.15.98:0/2701802024 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7fb1a401ad70 con 0x7fb1a8006890 2024-03-29T11:55:34.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.217+0000 7fb1977fe640 1 -- 172.21.15.98:0/2701802024 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7fb1a4084920 con 0x7fb1a8006890 2024-03-29T11:55:34.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.217+0000 7fb1977fe640 10 client.4473.objecter ms_dispatch 0x7fb1b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:55:34.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.217+0000 7fb1977fe640 3 client.4473.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:55:34.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.217+0000 7fb1977fe640 3 client.4473.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:55:34.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.217+0000 7fb1977fe640 20 client.4473.objecter dump_active .. 0 homeless 2024-03-29T11:55:34.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.218+0000 7fb1ae59c640 10 monclient: get_auth_request con 0x7fb19c06dd30 auth_method 0 2024-03-29T11:55:34.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.218+0000 7fb1977fe640 10 client.4473.objecter ms_handle_connect 0x7fb19c06dd30 2024-03-29T11:55:34.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.225+0000 7fb1977fe640 1 -- 172.21.15.98:0/2701802024 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb1a4050200 con 0x7fb1a8006890 2024-03-29T11:55:34.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.225+0000 7fb1977fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:55:34.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.225+0000 7fb1977fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:55:34.553 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.544+0000 7fb1b501e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:55:34.553 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.544+0000 7fb1b501e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:55:34.553 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.544+0000 7fb1b501e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:55:34.553 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.544+0000 7fb1b501e640 1 -- 172.21.15.98:0/2701802024 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1a8160d00 con 0x7fb1a8006890 2024-03-29T11:55:34.555 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:55:34.555 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:55:34.555 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.545+0000 7fb1977fe640 1 -- 172.21.15.98:0/2701802024 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7fb1a8160d00 con 0x7fb1a8006890 2024-03-29T11:55:34.555 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.545+0000 7fb1977fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:55:34.555 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.545+0000 7fb1977fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:55:34.557 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.548+0000 7fb1b501e640 10 librados: watch_flush enter 2024-03-29T11:55:34.558 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.548+0000 7fb1b501e640 10 librados: watch_flush exit 2024-03-29T11:55:34.558 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.548+0000 7fb1b501e640 20 client.4473.objecter shutdown clearing up homeless session... 2024-03-29T11:55:34.558 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.548+0000 7fb1b501e640 10 client.4473.objecter successfully canceled tick 2024-03-29T11:55:34.558 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.548+0000 7fb1b501e640 1 -- 172.21.15.98:0/2701802024 >> v1:172.21.15.98:6816/4065186974 conn(0x7fb19c06dd30 legacy=0x7fb19c0701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:34.558 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.548+0000 7fb1b501e640 10 monclient: shutdown 2024-03-29T11:55:34.558 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.548+0000 7fb1b501e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:55:34.558 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.548+0000 7fb1b501e640 1 -- 172.21.15.98:0/2701802024 >> v1:172.21.15.98:6789/0 conn(0x7fb1a8006890 legacy=0x7fb1a80c2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:34.558 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.548+0000 7fb1b501e640 1 -- 172.21.15.98:0/2701802024 shutdown_connections 2024-03-29T11:55:34.558 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.548+0000 7fb1b501e640 1 -- 172.21.15.98:0/2701802024 >> 172.21.15.98:0/2701802024 conn(0x7fb1a80058f0 msgr2=0x7fb1a8038130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:55:34.558 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.548+0000 7fb1b501e640 1 -- 172.21.15.98:0/2701802024 shutdown_connections 2024-03-29T11:55:34.558 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.549+0000 7fb1b501e640 1 -- 172.21.15.98:0/2701802024 wait complete. 2024-03-29T11:55:34.559 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:34.549+0000 7fb1b501e640 1 librados: shutdown 2024-03-29T11:55:34.575 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:55:41.382 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:55:41.542 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.533+0000 7f0e32354640 10 monclient: get_monmap_and_config 2024-03-29T11:55:41.542 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.533+0000 7f0e32354640 10 monclient: build_initial_monmap 2024-03-29T11:55:41.542 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.533+0000 7f0e32354640 10 monclient: monmap: 2024-03-29T11:55:41.542 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:55:41.542 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:55:41.542 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:55:41.534639+0000 2024-03-29T11:55:41.542 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:55:41.534639+0000 2024-03-29T11:55:41.543 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:55:41.543 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:41.543 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:55:41.543 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:55:41.543 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:55:41.543 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:41.544 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.535+0000 7f0e32354640 1 Processor -- start 2024-03-29T11:55:41.544 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.535+0000 7f0e32354640 1 -- start start 2024-03-29T11:55:41.544 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.535+0000 7f0e32354640 10 monclient: init 2024-03-29T11:55:41.545 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.536+0000 7f0e32354640 10 monclient: _reopen_session rank -1 2024-03-29T11:55:41.545 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.536+0000 7f0e32354640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:55:41.545 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.536+0000 7f0e32354640 10 monclient(hunting): picked mon.noname-c con 0x7f0e2c0b0770 addr v1:172.21.15.121:6789/0 2024-03-29T11:55:41.545 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.536+0000 7f0e32354640 10 monclient(hunting): picked mon.noname-a con 0x7f0e2c14a700 addr v1:172.21.15.98:6789/0 2024-03-29T11:55:41.545 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.536+0000 7f0e32354640 10 monclient(hunting): picked mon.noname-b con 0x7f0e2c0ac5a0 addr v1:172.21.15.98:6790/0 2024-03-29T11:55:41.545 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.536+0000 7f0e32354640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e2c0aba70 con 0x7f0e2c14a700 2024-03-29T11:55:41.545 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.536+0000 7f0e32354640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e2c0aecb0 con 0x7f0e2c0b0770 2024-03-29T11:55:41.545 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.536+0000 7f0e32354640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e2c0af150 con 0x7f0e2c0ac5a0 2024-03-29T11:55:41.545 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.536+0000 7f0e32354640 10 monclient(hunting): _renew_subs 2024-03-29T11:55:41.545 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.536+0000 7f0e32354640 10 monclient(hunting): authenticate will time out at 1248.621223s 2024-03-29T11:55:41.545 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.536+0000 7f0e2bfff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f0e2c0b0770 0x7f0e2c149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:44058/0 (socket says 172.21.15.98:44058) 2024-03-29T11:55:41.545 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.536+0000 7f0e2bfff640 1 -- 172.21.15.98:0/3165450352 learned_addr learned my addr 172.21.15.98:0/3165450352 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:55:41.546 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.537+0000 7f0e2affd640 1 -- 172.21.15.98:0/3165450352 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2069443631 0 0) 0x7f0e2c0aecb0 con 0x7f0e2c0b0770 2024-03-29T11:55:41.546 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.537+0000 7f0e2affd640 10 monclient(hunting): _negotiate 2024-03-29T11:55:41.546 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.537+0000 7f0e2affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:55:41.546 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.537+0000 7f0e2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:55:41.547 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.537+0000 7f0e2affd640 10 monclient(hunting): my global_id is 4489 2024-03-29T11:55:41.547 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.537+0000 7f0e2affd640 1 -- 172.21.15.98:0/3165450352 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e14003520 con 0x7f0e2c0b0770 2024-03-29T11:55:41.547 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.537+0000 7f0e2affd640 1 -- 172.21.15.98:0/3165450352 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2602527024 0 0) 0x7f0e2c0af150 con 0x7f0e2c0ac5a0 2024-03-29T11:55:41.547 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.537+0000 7f0e2affd640 10 monclient(hunting): _negotiate 2024-03-29T11:55:41.547 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.537+0000 7f0e2affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:55:41.547 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.537+0000 7f0e2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:55:41.547 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.537+0000 7f0e2affd640 10 monclient(hunting): my global_id is 4508 2024-03-29T11:55:41.547 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.537+0000 7f0e2affd640 1 -- 172.21.15.98:0/3165450352 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e2c0aecb0 con 0x7f0e2c0ac5a0 2024-03-29T11:55:41.547 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.537+0000 7f0e2affd640 1 -- 172.21.15.98:0/3165450352 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3820528760 0 0) 0x7f0e14003520 con 0x7f0e2c0b0770 2024-03-29T11:55:41.547 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.537+0000 7f0e2affd640 1 -- 172.21.15.98:0/3165450352 >> v1:172.21.15.98:6790/0 conn(0x7f0e2c0ac5a0 legacy=0x7f0e2c151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:41.547 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.537+0000 7f0e2affd640 1 -- 172.21.15.98:0/3165450352 >> v1:172.21.15.98:6789/0 conn(0x7f0e2c14a700 legacy=0x7f0e2c0abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:55:41.548 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.539+0000 7f0e2affd640 10 monclient: _finish_hunting 0 2024-03-29T11:55:41.548 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.539+0000 7f0e2affd640 1 monclient: found mon.noname-c 2024-03-29T11:55:41.548 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.539+0000 7f0e2affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:55:41.548 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.539+0000 7f0e2affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:55:41.548 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.539+0000 7f0e2affd640 1 -- 172.21.15.98:0/3165450352 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e2c0ac9d0 con 0x7f0e2c0b0770 2024-03-29T11:55:41.548 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.539+0000 7f0e2affd640 10 monclient: _finish_auth 0 2024-03-29T11:55:41.548 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.539+0000 7f0e2affd640 10 monclient: _check_auth_tickets 2024-03-29T11:55:41.548 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.539+0000 7f0e2affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:55:41.548 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.539+0000 7f0e32354640 5 monclient: authenticate success, global_id 4489 2024-03-29T11:55:41.548 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.539+0000 7f0e32354640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:55:41.548 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.539+0000 7f0e2affd640 1 -- 172.21.15.98:0/3165450352 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f0e1c0016b0 con 0x7f0e2c0b0770 2024-03-29T11:55:41.548 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.539+0000 7f0e2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:55:41.548 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.539+0000 7f0e2affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:55:41.548 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.539+0000 7f0e2affd640 10 monclient: dump: 2024-03-29T11:55:41.549 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:55:41.549 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:55:41.549 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:41.549 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:41.549 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:55:41.549 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:41.549 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:55:41.549 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:55:41.549 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:55:41.549 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:41.549 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.539+0000 7f0e32354640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:55:41.550 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.540+0000 7f0e2affd640 1 -- 172.21.15.98:0/3165450352 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e1c001810 con 0x7f0e2c0b0770 2024-03-29T11:55:41.550 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.540+0000 7f0e2affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:55:41.550 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.540+0000 7f0e2affd640 1 -- 172.21.15.98:0/3165450352 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f0e1c001c20 con 0x7f0e2c0b0770 2024-03-29T11:55:41.550 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.540+0000 7f0e2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:55:41.550 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.540+0000 7f0e2affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:55:41.551 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.540+0000 7f0e2affd640 10 monclient: dump: 2024-03-29T11:55:41.551 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:55:41.551 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:55:41.551 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:41.551 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:41.551 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:55:41.551 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:41.551 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:55:41.551 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:55:41.551 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:55:41.551 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:41.551 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.540+0000 7f0e32354640 10 monclient: get_monmap_and_config success 2024-03-29T11:55:41.551 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.540+0000 7f0e32354640 4 set_mon_vals no callback set 2024-03-29T11:55:41.551 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.542+0000 7f0e32354640 10 monclient: shutdown 2024-03-29T11:55:41.551 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.542+0000 7f0e32354640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:55:41.552 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.542+0000 7f0e32354640 1 -- 172.21.15.98:0/3165450352 >> v1:172.21.15.121:6789/0 conn(0x7f0e2c0b0770 legacy=0x7f0e2c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:41.552 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.542+0000 7f0e32354640 1 -- 172.21.15.98:0/3165450352 shutdown_connections 2024-03-29T11:55:41.552 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.542+0000 7f0e32354640 1 -- 172.21.15.98:0/3165450352 >> 172.21.15.98:0/3165450352 conn(0x7f0e2c06b660 msgr2=0x7f0e2c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:55:41.552 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.542+0000 7f0e32354640 1 -- 172.21.15.98:0/3165450352 shutdown_connections 2024-03-29T11:55:41.553 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.542+0000 7f0e32354640 1 -- 172.21.15.98:0/3165450352 wait complete. 2024-03-29T11:55:41.553 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.543+0000 7f0e32354640 10 monclient: build_initial_monmap 2024-03-29T11:55:41.553 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.543+0000 7f0e32354640 10 monclient: monmap: 2024-03-29T11:55:41.553 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:55:41.553 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:55:41.553 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:55:41.553 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:55:41.553 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:55:41.553 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:41.553 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:55:41.553 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:55:41.553 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:55:41.553 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:41.554 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.543+0000 7f0e32354640 1 librados: starting msgr at 2024-03-29T11:55:41.554 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.543+0000 7f0e32354640 1 librados: starting objecter 2024-03-29T11:55:41.554 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.544+0000 7f0e32354640 1 Processor -- start 2024-03-29T11:55:41.554 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.544+0000 7f0e32354640 1 -- start start 2024-03-29T11:55:41.554 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.544+0000 7f0e32354640 1 librados: setting wanted keys 2024-03-29T11:55:41.554 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.544+0000 7f0e32354640 1 librados: calling monclient init 2024-03-29T11:55:41.554 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.544+0000 7f0e32354640 10 monclient: init 2024-03-29T11:55:41.556 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.545+0000 7f0e32354640 10 monclient: _reopen_session rank -1 2024-03-29T11:55:41.556 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.545+0000 7f0e32354640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:55:41.556 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.545+0000 7f0e32354640 10 monclient(hunting): picked mon.noname-a con 0x7f0e2c0ac5a0 addr v1:172.21.15.98:6789/0 2024-03-29T11:55:41.556 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.545+0000 7f0e32354640 10 monclient(hunting): picked mon.noname-c con 0x7f0e2c0b0770 addr v1:172.21.15.98:6790/0 2024-03-29T11:55:41.556 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.545+0000 7f0e32354640 10 monclient(hunting): picked mon.noname-b con 0x7f0e2c14a700 addr v1:172.21.15.121:6789/0 2024-03-29T11:55:41.556 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.545+0000 7f0e32354640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e2c1272d0 con 0x7f0e2c0ac5a0 2024-03-29T11:55:41.556 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.545+0000 7f0e32354640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e2c128010 con 0x7f0e2c14a700 2024-03-29T11:55:41.556 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.545+0000 7f0e32354640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e2c125c20 con 0x7f0e2c0b0770 2024-03-29T11:55:41.556 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.545+0000 7f0e32354640 10 monclient(hunting): _renew_subs 2024-03-29T11:55:41.556 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.545+0000 7f0e32354640 10 monclient(hunting): authenticate will time out at 1248.630853s 2024-03-29T11:55:41.556 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.545+0000 7f0e2b7fe640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f0e2c0b0770 0x7f0e2c1db820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:39866/0 (socket says 172.21.15.98:39866) 2024-03-29T11:55:41.556 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.545+0000 7f0e2b7fe640 1 -- 172.21.15.98:0/1725895672 learned_addr learned my addr 172.21.15.98:0/1725895672 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:55:41.557 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.546+0000 7f0e28ff9640 10 client.?.objecter ms_handle_connect 0x7f0e2c0b0770 2024-03-29T11:55:41.557 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.546+0000 7f0e28ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:55:41.557 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.546+0000 7f0e28ff9640 1 -- 172.21.15.98:0/1725895672 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 168975118 0 0) 0x7f0e2c125c20 con 0x7f0e2c0b0770 2024-03-29T11:55:41.557 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.546+0000 7f0e28ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:55:41.557 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.546+0000 7f0e28ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:55:41.557 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.546+0000 7f0e28ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:55:41.557 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.546+0000 7f0e28ff9640 10 monclient(hunting): my global_id is 4511 2024-03-29T11:55:41.557 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.546+0000 7f0e28ff9640 1 -- 172.21.15.98:0/1725895672 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e18004eb0 con 0x7f0e2c0b0770 2024-03-29T11:55:41.560 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.548+0000 7f0e28ff9640 10 client.?.objecter ms_handle_connect 0x7f0e2c14a700 2024-03-29T11:55:41.560 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.548+0000 7f0e28ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:55:41.560 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.548+0000 7f0e28ff9640 1 -- 172.21.15.98:0/1725895672 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1528769197 0 0) 0x7f0e18004eb0 con 0x7f0e2c0b0770 2024-03-29T11:55:41.560 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.548+0000 7f0e28ff9640 1 -- 172.21.15.98:0/1725895672 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e2c125c20 con 0x7f0e2c0b0770 2024-03-29T11:55:41.561 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.548+0000 7f0e28ff9640 1 -- 172.21.15.98:0/1725895672 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f0e24001b50 con 0x7f0e2c0b0770 2024-03-29T11:55:41.561 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.548+0000 7f0e28ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:55:41.561 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.548+0000 7f0e28ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:55:41.561 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.548+0000 7f0e28ff9640 10 monclient(hunting): dump: 2024-03-29T11:55:41.561 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:55:41.561 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:55:41.561 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:41.561 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:41.561 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:55:41.561 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:41.561 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:55:41.561 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:55:41.561 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:55:41.561 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:41.561 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.548+0000 7f0e28ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:55:41.562 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.548+0000 7f0e28ff9640 1 -- 172.21.15.98:0/1725895672 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2909942542 0 0) 0x7f0e2c128010 con 0x7f0e2c14a700 2024-03-29T11:55:41.562 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.548+0000 7f0e28ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:55:41.562 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.548+0000 7f0e28ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:55:41.562 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.548+0000 7f0e28ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:55:41.562 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.548+0000 7f0e28ff9640 10 monclient(hunting): my global_id is 4492 2024-03-29T11:55:41.562 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.548+0000 7f0e28ff9640 1 -- 172.21.15.98:0/1725895672 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e18004eb0 con 0x7f0e2c14a700 2024-03-29T11:55:41.562 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.548+0000 7f0e28ff9640 1 -- 172.21.15.98:0/1725895672 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4230837907 0 0) 0x7f0e2c125c20 con 0x7f0e2c0b0770 2024-03-29T11:55:41.562 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.548+0000 7f0e28ff9640 1 -- 172.21.15.98:0/1725895672 >> v1:172.21.15.121:6789/0 conn(0x7f0e2c14a700 legacy=0x7f0e2c1e6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:41.562 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.548+0000 7f0e28ff9640 1 -- 172.21.15.98:0/1725895672 >> v1:172.21.15.98:6789/0 conn(0x7f0e2c0ac5a0 legacy=0x7f0e2c144c00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:55:41.562 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.549+0000 7f0e28ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:55:41.562 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.549+0000 7f0e28ff9640 1 monclient: found mon.c 2024-03-29T11:55:41.562 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.549+0000 7f0e28ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:55:41.562 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.549+0000 7f0e28ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:55:41.563 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.549+0000 7f0e28ff9640 1 -- 172.21.15.98:0/1725895672 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e2c1265e0 con 0x7f0e2c0b0770 2024-03-29T11:55:41.563 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.549+0000 7f0e28ff9640 10 monclient: _finish_auth 0 2024-03-29T11:55:41.563 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.549+0000 7f0e28ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:55:41.563 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.549+0000 7f0e28ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:55:41.563 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.549+0000 7f0e32354640 5 monclient: authenticate success, global_id 4511 2024-03-29T11:55:41.563 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.549+0000 7f0e32354640 10 monclient: _renew_subs 2024-03-29T11:55:41.563 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.549+0000 7f0e32354640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:55:41.563 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.549+0000 7f0e32354640 1 -- 172.21.15.98:0/1725895672 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e2c145910 con 0x7f0e2c0b0770 2024-03-29T11:55:41.563 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.549+0000 7f0e32354640 10 client.4511.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:55:41.563 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.549+0000 7f0e32354640 10 monclient: _renew_subs 2024-03-29T11:55:41.563 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.549+0000 7f0e32354640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:55:41.563 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.549+0000 7f0e32354640 1 -- 172.21.15.98:0/1725895672 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e2c1454f0 con 0x7f0e2c0b0770 2024-03-29T11:55:41.563 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.549+0000 7f0e32354640 1 librados: init done 2024-03-29T11:55:41.563 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.550+0000 7f0e28ff9640 1 -- 172.21.15.98:0/1725895672 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e240020e0 con 0x7f0e2c0b0770 2024-03-29T11:55:41.563 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.550+0000 7f0e28ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:55:41.564 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.550+0000 7f0e28ff9640 1 -- 172.21.15.98:0/1725895672 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f0e24004d80 con 0x7f0e2c0b0770 2024-03-29T11:55:41.564 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.550+0000 7f0e28ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:55:41.564 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.550+0000 7f0e28ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:55:41.564 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.550+0000 7f0e28ff9640 10 monclient: dump: 2024-03-29T11:55:41.564 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:55:41.564 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:55:41.564 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:41.564 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:41.564 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:55:41.564 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:41.564 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:55:41.564 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:55:41.564 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:55:41.564 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:41.564 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.550+0000 7f0e32354640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:55:41.564 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.550+0000 7f0e32354640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:55:41.564 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.551+0000 7f0e32354640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:55:41.565 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.551+0000 7f0e2a7fc640 4 set_mon_vals no callback set 2024-03-29T11:55:41.565 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.551+0000 7f0e32354640 1 -- 172.21.15.98:0/1725895672 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e2c0ab560 con 0x7f0e2c0b0770 2024-03-29T11:55:41.565 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.553+0000 7f0e28ff9640 1 -- 172.21.15.98:0/1725895672 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f0e2401ad50 con 0x7f0e2c0b0770 2024-03-29T11:55:41.565 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.554+0000 7f0e28ff9640 1 -- 172.21.15.98:0/1725895672 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f0e24084d40 con 0x7f0e2c0b0770 2024-03-29T11:55:41.565 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.554+0000 7f0e28ff9640 10 client.4511.objecter ms_dispatch 0x7f0e2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:55:41.565 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.554+0000 7f0e28ff9640 3 client.4511.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:55:41.565 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.554+0000 7f0e28ff9640 3 client.4511.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:55:41.566 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.554+0000 7f0e28ff9640 20 client.4511.objecter dump_active .. 0 homeless 2024-03-29T11:55:41.566 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.555+0000 7f0e2bfff640 10 monclient: get_auth_request con 0x7f0e1806dd90 auth_method 0 2024-03-29T11:55:41.566 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.555+0000 7f0e28ff9640 10 client.4511.objecter ms_handle_connect 0x7f0e1806dd90 2024-03-29T11:55:41.572 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.564+0000 7f0e28ff9640 1 -- 172.21.15.98:0/1725895672 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0e24050450 con 0x7f0e2c0b0770 2024-03-29T11:55:41.573 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.564+0000 7f0e28ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:55:41.573 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.564+0000 7f0e28ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:55:41.933 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.925+0000 7f0e32354640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:55:41.933 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.925+0000 7f0e32354640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:55:41.934 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.925+0000 7f0e32354640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:55:41.934 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.925+0000 7f0e32354640 1 -- 172.21.15.98:0/1725895672 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e2c075f10 con 0x7f0e2c0b0770 2024-03-29T11:55:41.935 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:55:41.935 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:55:41.935 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.926+0000 7f0e28ff9640 1 -- 172.21.15.98:0/1725895672 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f0e24052550 con 0x7f0e2c0b0770 2024-03-29T11:55:41.935 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.926+0000 7f0e28ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:55:41.935 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.926+0000 7f0e28ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:55:41.938 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.929+0000 7f0e32354640 10 librados: watch_flush enter 2024-03-29T11:55:41.938 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.929+0000 7f0e32354640 10 librados: watch_flush exit 2024-03-29T11:55:41.938 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.929+0000 7f0e32354640 20 client.4511.objecter shutdown clearing up homeless session... 2024-03-29T11:55:41.938 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.929+0000 7f0e32354640 10 client.4511.objecter successfully canceled tick 2024-03-29T11:55:41.938 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.930+0000 7f0e32354640 1 -- 172.21.15.98:0/1725895672 >> v1:172.21.15.98:6816/4065186974 conn(0x7f0e1806dd90 legacy=0x7f0e18070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:41.938 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.930+0000 7f0e32354640 10 monclient: shutdown 2024-03-29T11:55:41.938 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.930+0000 7f0e32354640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:55:41.938 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.930+0000 7f0e32354640 1 -- 172.21.15.98:0/1725895672 >> v1:172.21.15.98:6790/0 conn(0x7f0e2c0b0770 legacy=0x7f0e2c1db820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:41.938 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.930+0000 7f0e32354640 1 -- 172.21.15.98:0/1725895672 shutdown_connections 2024-03-29T11:55:41.939 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.930+0000 7f0e32354640 1 -- 172.21.15.98:0/1725895672 >> 172.21.15.98:0/1725895672 conn(0x7f0e2c06b660 msgr2=0x7f0e2c0b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:55:41.939 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.930+0000 7f0e32354640 1 -- 172.21.15.98:0/1725895672 shutdown_connections 2024-03-29T11:55:41.939 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.930+0000 7f0e32354640 1 -- 172.21.15.98:0/1725895672 wait complete. 2024-03-29T11:55:41.939 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:41.930+0000 7f0e32354640 1 librados: shutdown 2024-03-29T11:55:41.955 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:55:48.762 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:55:48.907 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.898+0000 7fdcaf59e640 10 monclient: get_monmap_and_config 2024-03-29T11:55:48.907 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.898+0000 7fdcaf59e640 10 monclient: build_initial_monmap 2024-03-29T11:55:48.907 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.898+0000 7fdcaf59e640 10 monclient: monmap: 2024-03-29T11:55:48.907 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:55:48.908 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:55:48.908 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:55:48.899548+0000 2024-03-29T11:55:48.908 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:55:48.899548+0000 2024-03-29T11:55:48.908 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:55:48.908 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:48.908 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:55:48.908 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:55:48.908 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:55:48.908 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:48.911 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.899+0000 7fdcaf59e640 1 Processor -- start 2024-03-29T11:55:48.911 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.899+0000 7fdcaf59e640 1 -- start start 2024-03-29T11:55:48.911 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.899+0000 7fdcaf59e640 10 monclient: init 2024-03-29T11:55:48.911 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.900+0000 7fdcaf59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:55:48.911 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.901+0000 7fdcaf59e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:55:48.911 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.901+0000 7fdcaf59e640 10 monclient(hunting): picked mon.noname-c con 0x7fdca80389e0 addr v1:172.21.15.121:6789/0 2024-03-29T11:55:48.911 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.901+0000 7fdcaf59e640 10 monclient(hunting): picked mon.noname-b con 0x7fdca8038dc0 addr v1:172.21.15.98:6790/0 2024-03-29T11:55:48.911 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.901+0000 7fdcaf59e640 10 monclient(hunting): picked mon.noname-a con 0x7fdca8030660 addr v1:172.21.15.98:6789/0 2024-03-29T11:55:48.911 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.901+0000 7fdcaf59e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca80c54c0 con 0x7fdca8030660 2024-03-29T11:55:48.911 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.901+0000 7fdcaf59e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca8036430 con 0x7fdca80389e0 2024-03-29T11:55:48.911 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.901+0000 7fdcaf59e640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca8039840 con 0x7fdca8038dc0 2024-03-29T11:55:48.911 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.901+0000 7fdcaf59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:55:48.912 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.901+0000 7fdcaf59e640 10 monclient(hunting): authenticate will time out at 1255.986253s 2024-03-29T11:55:48.912 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.901+0000 7fdcaed9d640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fdca8030660 0x7fdca80d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:60606/0 (socket says 172.21.15.98:60606) 2024-03-29T11:55:48.912 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.901+0000 7fdcaed9d640 1 -- 172.21.15.98:0/4274300590 learned_addr learned my addr 172.21.15.98:0/4274300590 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:55:48.912 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.901+0000 7fdcad59a640 1 -- 172.21.15.98:0/4274300590 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3246403762 0 0) 0x7fdca80c54c0 con 0x7fdca8030660 2024-03-29T11:55:48.912 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.901+0000 7fdcad59a640 10 monclient(hunting): _negotiate 2024-03-29T11:55:48.912 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.901+0000 7fdcad59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:55:48.912 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.901+0000 7fdcad59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:55:48.912 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.901+0000 7fdcad59a640 10 monclient(hunting): my global_id is 4476 2024-03-29T11:55:48.912 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.901+0000 7fdcad59a640 1 -- 172.21.15.98:0/4274300590 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc98003520 con 0x7fdca8030660 2024-03-29T11:55:48.912 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.902+0000 7fdcad59a640 1 -- 172.21.15.98:0/4274300590 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4210500225 0 0) 0x7fdc98003520 con 0x7fdca8030660 2024-03-29T11:55:48.912 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.902+0000 7fdcad59a640 1 -- 172.21.15.98:0/4274300590 >> v1:172.21.15.98:6790/0 conn(0x7fdca8038dc0 legacy=0x7fdca80cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:48.912 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.902+0000 7fdcad59a640 1 -- 172.21.15.98:0/4274300590 >> v1:172.21.15.121:6789/0 conn(0x7fdca80389e0 legacy=0x7fdca8033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:48.912 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.902+0000 7fdcad59a640 10 monclient: _finish_hunting 0 2024-03-29T11:55:48.912 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.902+0000 7fdcad59a640 1 monclient: found mon.noname-a 2024-03-29T11:55:48.912 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.902+0000 7fdcad59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:55:48.913 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.902+0000 7fdcad59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T11:55:48.913 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.902+0000 7fdcad59a640 1 -- 172.21.15.98:0/4274300590 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca8035690 con 0x7fdca8030660 2024-03-29T11:55:48.913 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.902+0000 7fdcad59a640 10 monclient: _finish_auth 0 2024-03-29T11:55:48.913 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.902+0000 7fdcad59a640 10 monclient: _check_auth_tickets 2024-03-29T11:55:48.913 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.902+0000 7fdcad59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:55:48.913 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.902+0000 7fdcad59a640 1 -- 172.21.15.98:0/4274300590 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fdcb004f730 con 0x7fdca8030660 2024-03-29T11:55:48.913 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.902+0000 7fdcad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:55:48.913 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.902+0000 7fdcad59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:55:48.913 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.902+0000 7fdcad59a640 10 monclient: dump: 2024-03-29T11:55:48.913 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:55:48.913 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:55:48.913 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:48.913 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:48.913 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:55:48.913 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:48.914 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:55:48.914 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:55:48.914 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:55:48.914 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:48.914 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.902+0000 7fdcaf59e640 5 monclient: authenticate success, global_id 4476 2024-03-29T11:55:48.914 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.902+0000 7fdcaf59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:55:48.916 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.903+0000 7fdcad59a640 1 -- 172.21.15.98:0/4274300590 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdcb0075f10 con 0x7fdca8030660 2024-03-29T11:55:48.916 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.903+0000 7fdcad59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:55:48.916 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.903+0000 7fdcad59a640 1 -- 172.21.15.98:0/4274300590 <== mon.0 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fdcb00758b0 con 0x7fdca8030660 2024-03-29T11:55:48.916 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.903+0000 7fdcad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:55:48.917 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.903+0000 7fdcad59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:55:48.917 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.903+0000 7fdcad59a640 10 monclient: dump: 2024-03-29T11:55:48.917 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:55:48.917 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:55:48.917 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:48.917 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:48.917 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:55:48.917 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:48.917 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:55:48.917 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:55:48.917 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:55:48.917 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:48.917 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.903+0000 7fdcaf59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:55:48.917 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.903+0000 7fdcaf59e640 4 set_mon_vals no callback set 2024-03-29T11:55:48.917 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.904+0000 7fdcaf59e640 10 monclient: shutdown 2024-03-29T11:55:48.918 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.904+0000 7fdcaf59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:55:48.918 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.904+0000 7fdcaf59e640 1 -- 172.21.15.98:0/4274300590 >> v1:172.21.15.98:6789/0 conn(0x7fdca8030660 legacy=0x7fdca80d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:48.918 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.904+0000 7fdcaf59e640 1 -- 172.21.15.98:0/4274300590 shutdown_connections 2024-03-29T11:55:48.918 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.904+0000 7fdcaf59e640 1 -- 172.21.15.98:0/4274300590 >> 172.21.15.98:0/4274300590 conn(0x7fdca8005c00 msgr2=0x7fdca8006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:55:48.918 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.904+0000 7fdcaf59e640 1 -- 172.21.15.98:0/4274300590 shutdown_connections 2024-03-29T11:55:48.918 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.904+0000 7fdcaf59e640 1 -- 172.21.15.98:0/4274300590 wait complete. 2024-03-29T11:55:48.918 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.905+0000 7fdcaf59e640 10 monclient: build_initial_monmap 2024-03-29T11:55:48.918 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.906+0000 7fdcaf59e640 10 monclient: monmap: 2024-03-29T11:55:48.918 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:55:48.918 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:55:48.918 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:55:48.918 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:55:48.918 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:55:48.918 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:48.918 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:55:48.919 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:55:48.919 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:55:48.919 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:48.919 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.906+0000 7fdcaf59e640 1 librados: starting msgr at 2024-03-29T11:55:48.919 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.906+0000 7fdcaf59e640 1 librados: starting objecter 2024-03-29T11:55:48.919 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.906+0000 7fdcaf59e640 1 Processor -- start 2024-03-29T11:55:48.919 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.907+0000 7fdcaf59e640 1 -- start start 2024-03-29T11:55:48.919 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.907+0000 7fdcaf59e640 1 librados: setting wanted keys 2024-03-29T11:55:48.919 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.907+0000 7fdcaf59e640 1 librados: calling monclient init 2024-03-29T11:55:48.919 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.907+0000 7fdcaf59e640 10 monclient: init 2024-03-29T11:55:48.919 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.908+0000 7fdcaf59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:55:48.919 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.908+0000 7fdcaf59e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:55:48.919 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.908+0000 7fdcaf59e640 10 monclient(hunting): picked mon.noname-b con 0x7fdca8030660 addr v1:172.21.15.121:6789/0 2024-03-29T11:55:48.919 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.908+0000 7fdcae59c640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fdca8030660 0x7fdca815cc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:34974/0 (socket says 172.21.15.98:34974) 2024-03-29T11:55:48.920 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.908+0000 7fdcae59c640 1 -- 172.21.15.98:0/3050560752 learned_addr learned my addr 172.21.15.98:0/3050560752 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:55:48.920 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.908+0000 7fdcaf59e640 10 monclient(hunting): picked mon.noname-c con 0x7fdca80389e0 addr v1:172.21.15.98:6790/0 2024-03-29T11:55:48.925 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.909+0000 7fdcaf59e640 10 monclient(hunting): picked mon.noname-a con 0x7fdca80c8100 addr v1:172.21.15.98:6789/0 2024-03-29T11:55:48.925 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.909+0000 7fdcaf59e640 1 -- 172.21.15.98:0/3050560752 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca80c73d0 con 0x7fdca80c8100 2024-03-29T11:55:48.926 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.909+0000 7fdcaf59e640 1 -- 172.21.15.98:0/3050560752 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca8030040 con 0x7fdca8030660 2024-03-29T11:55:48.926 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.909+0000 7fdc9effd640 10 client.?.objecter ms_handle_connect 0x7fdca80389e0 2024-03-29T11:55:48.926 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.909+0000 7fdc9effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:55:48.926 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.909+0000 7fdc9effd640 10 client.?.objecter ms_handle_connect 0x7fdca80c8100 2024-03-29T11:55:48.926 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.909+0000 7fdc9effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:55:48.926 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.909+0000 7fdc9effd640 1 -- 172.21.15.98:0/3050560752 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1465379780 0 0) 0x7fdca80c73d0 con 0x7fdca80c8100 2024-03-29T11:55:48.926 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.909+0000 7fdcaf59e640 1 -- 172.21.15.98:0/3050560752 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca80c7c00 con 0x7fdca80389e0 2024-03-29T11:55:48.926 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.910+0000 7fdcaf59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:55:48.926 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.910+0000 7fdcaf59e640 10 monclient(hunting): authenticate will time out at 1255.995258s 2024-03-29T11:55:48.926 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.910+0000 7fdc9effd640 10 monclient(hunting): _negotiate 2024-03-29T11:55:48.926 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.910+0000 7fdc9effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:55:48.926 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.910+0000 7fdc9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:55:48.926 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.910+0000 7fdc9effd640 10 monclient(hunting): my global_id is 4479 2024-03-29T11:55:48.927 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.910+0000 7fdc9effd640 1 -- 172.21.15.98:0/3050560752 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc90004eb0 con 0x7fdca80c8100 2024-03-29T11:55:48.927 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.910+0000 7fdc9effd640 1 -- 172.21.15.98:0/3050560752 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1793749586 0 0) 0x7fdca80c7c00 con 0x7fdca80389e0 2024-03-29T11:55:48.927 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.910+0000 7fdc9effd640 10 monclient(hunting): _negotiate 2024-03-29T11:55:48.927 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.910+0000 7fdc9effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:55:48.927 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.910+0000 7fdc9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:55:48.927 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.910+0000 7fdc9effd640 10 monclient(hunting): my global_id is 4517 2024-03-29T11:55:48.927 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.910+0000 7fdc9effd640 1 -- 172.21.15.98:0/3050560752 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdca80c73d0 con 0x7fdca80389e0 2024-03-29T11:55:48.927 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.911+0000 7fdc9effd640 10 client.?.objecter ms_handle_connect 0x7fdca8030660 2024-03-29T11:55:48.927 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.911+0000 7fdc9effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:55:48.927 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.911+0000 7fdc9effd640 1 -- 172.21.15.98:0/3050560752 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1341087142 0 0) 0x7fdc90004eb0 con 0x7fdca80c8100 2024-03-29T11:55:48.927 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.911+0000 7fdc9effd640 1 -- 172.21.15.98:0/3050560752 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdca80c7c00 con 0x7fdca80c8100 2024-03-29T11:55:48.927 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.911+0000 7fdc9effd640 1 -- 172.21.15.98:0/3050560752 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fdcb004f400 con 0x7fdca80c8100 2024-03-29T11:55:48.927 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.911+0000 7fdc9effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:55:48.927 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.911+0000 7fdc9effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:55:48.927 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.911+0000 7fdc9effd640 10 monclient(hunting): dump: 2024-03-29T11:55:48.928 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:55:48.928 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:55:48.928 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:48.928 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:48.928 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:55:48.928 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:48.928 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:55:48.928 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:55:48.928 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:55:48.928 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:48.928 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.911+0000 7fdc9effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:55:48.928 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.911+0000 7fdc9effd640 1 -- 172.21.15.98:0/3050560752 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3407007532 0 0) 0x7fdca8030040 con 0x7fdca8030660 2024-03-29T11:55:48.928 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.911+0000 7fdc9effd640 10 monclient(hunting): _negotiate 2024-03-29T11:55:48.928 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.911+0000 7fdc9effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:55:48.928 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.911+0000 7fdc9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:55:48.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.911+0000 7fdc9effd640 10 monclient(hunting): my global_id is 4498 2024-03-29T11:55:48.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.911+0000 7fdc9effd640 1 -- 172.21.15.98:0/3050560752 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc90004eb0 con 0x7fdca8030660 2024-03-29T11:55:48.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.911+0000 7fdc9effd640 1 -- 172.21.15.98:0/3050560752 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2672055741 0 0) 0x7fdca80c7c00 con 0x7fdca80c8100 2024-03-29T11:55:48.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.911+0000 7fdc9effd640 1 -- 172.21.15.98:0/3050560752 >> v1:172.21.15.98:6790/0 conn(0x7fdca80389e0 legacy=0x7fdca80c5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:48.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.911+0000 7fdc9effd640 1 -- 172.21.15.98:0/3050560752 >> v1:172.21.15.121:6789/0 conn(0x7fdca8030660 legacy=0x7fdca815cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:48.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.912+0000 7fdc9effd640 10 monclient: _finish_hunting 0 2024-03-29T11:55:48.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.912+0000 7fdc9effd640 1 monclient: found mon.a 2024-03-29T11:55:48.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.912+0000 7fdc9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:55:48.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.912+0000 7fdc9effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:55:48.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.912+0000 7fdc9effd640 1 -- 172.21.15.98:0/3050560752 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca80c8590 con 0x7fdca80c8100 2024-03-29T11:55:48.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.912+0000 7fdc9effd640 10 monclient: _finish_auth 0 2024-03-29T11:55:48.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.912+0000 7fdc9effd640 10 monclient: _check_auth_tickets 2024-03-29T11:55:48.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.912+0000 7fdc9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:55:48.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.912+0000 7fdcaf59e640 5 monclient: authenticate success, global_id 4479 2024-03-29T11:55:48.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.912+0000 7fdcaf59e640 10 monclient: _renew_subs 2024-03-29T11:55:48.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.912+0000 7fdcaf59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:55:48.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.912+0000 7fdcaf59e640 1 -- 172.21.15.98:0/3050560752 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdca8159cc0 con 0x7fdca80c8100 2024-03-29T11:55:48.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.912+0000 7fdcaf59e640 10 client.4479.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:55:48.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.912+0000 7fdcaf59e640 10 monclient: _renew_subs 2024-03-29T11:55:48.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.912+0000 7fdcaf59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:55:48.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.912+0000 7fdcaf59e640 1 -- 172.21.15.98:0/3050560752 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdca815a1a0 con 0x7fdca80c8100 2024-03-29T11:55:48.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.912+0000 7fdcaf59e640 1 librados: init done 2024-03-29T11:55:48.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.913+0000 7fdc9effd640 1 -- 172.21.15.98:0/3050560752 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdcb004fdd0 con 0x7fdca80c8100 2024-03-29T11:55:48.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.913+0000 7fdc9effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:55:48.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.913+0000 7fdc9effd640 1 -- 172.21.15.98:0/3050560752 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fdcb0067bf0 con 0x7fdca80c8100 2024-03-29T11:55:48.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.913+0000 7fdc9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:55:48.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.913+0000 7fdc9effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:55:48.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.913+0000 7fdc9effd640 10 monclient: dump: 2024-03-29T11:55:48.930 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:55:48.931 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:55:48.931 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:48.931 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:48.931 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:55:48.931 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:48.931 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:55:48.931 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:55:48.931 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:55:48.931 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:48.931 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.913+0000 7fdcacd99640 4 set_mon_vals no callback set 2024-03-29T11:55:48.931 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.913+0000 7fdcaf59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:55:48.931 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.914+0000 7fdcaf59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:55:48.931 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.914+0000 7fdcaf59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:55:48.931 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.914+0000 7fdcaf59e640 1 -- 172.21.15.98:0/3050560752 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc94005080 con 0x7fdca80c8100 2024-03-29T11:55:48.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.915+0000 7fdc9effd640 1 -- 172.21.15.98:0/3050560752 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7fdcb0067d50 con 0x7fdca80c8100 2024-03-29T11:55:48.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.916+0000 7fdcae59c640 10 monclient: get_auth_request con 0x7fdc9006db10 auth_method 0 2024-03-29T11:55:48.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.916+0000 7fdc9effd640 1 -- 172.21.15.98:0/3050560752 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7fdcb00ff3a0 con 0x7fdca80c8100 2024-03-29T11:55:48.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.916+0000 7fdc9effd640 10 client.4479.objecter ms_dispatch 0x7fdcb0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:55:48.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.916+0000 7fdc9effd640 3 client.4479.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:55:48.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.916+0000 7fdc9effd640 3 client.4479.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:55:48.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.916+0000 7fdc9effd640 20 client.4479.objecter dump_active .. 0 homeless 2024-03-29T11:55:48.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.916+0000 7fdc9effd640 10 client.4479.objecter ms_handle_connect 0x7fdc9006db10 2024-03-29T11:55:48.933 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.924+0000 7fdc9effd640 1 -- 172.21.15.98:0/3050560752 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdcb00cac80 con 0x7fdca80c8100 2024-03-29T11:55:48.933 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.924+0000 7fdc9effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:55:48.933 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:48.924+0000 7fdc9effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:55:49.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:49.289+0000 7fdcaf59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:55:49.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:49.289+0000 7fdcaf59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:55:49.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:49.289+0000 7fdcaf59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:55:49.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:49.289+0000 7fdcaf59e640 1 -- 172.21.15.98:0/3050560752 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc94005330 con 0x7fdca80c8100 2024-03-29T11:55:49.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:49.290+0000 7fdc9effd640 1 -- 172.21.15.98:0/3050560752 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7fdcb00ccbb0 con 0x7fdca80c8100 2024-03-29T11:55:49.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:49.290+0000 7fdc9effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:55:49.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:49.290+0000 7fdc9effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:55:49.300 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:55:49.300 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:55:49.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:49.294+0000 7fdcaf59e640 10 librados: watch_flush enter 2024-03-29T11:55:49.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:49.294+0000 7fdcaf59e640 10 librados: watch_flush exit 2024-03-29T11:55:49.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:49.294+0000 7fdcaf59e640 20 client.4479.objecter shutdown clearing up homeless session... 2024-03-29T11:55:49.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:49.294+0000 7fdcaf59e640 10 client.4479.objecter successfully canceled tick 2024-03-29T11:55:49.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:49.294+0000 7fdcaf59e640 1 -- 172.21.15.98:0/3050560752 >> v1:172.21.15.98:6816/4065186974 conn(0x7fdc9006db10 legacy=0x7fdc9006ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:49.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:49.294+0000 7fdcaf59e640 10 monclient: shutdown 2024-03-29T11:55:49.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:49.294+0000 7fdcaf59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:55:49.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:49.294+0000 7fdcaf59e640 1 -- 172.21.15.98:0/3050560752 >> v1:172.21.15.98:6789/0 conn(0x7fdca80c8100 legacy=0x7fdca80c5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:49.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:49.294+0000 7fdcaf59e640 1 -- 172.21.15.98:0/3050560752 shutdown_connections 2024-03-29T11:55:49.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:49.294+0000 7fdcaf59e640 1 -- 172.21.15.98:0/3050560752 >> 172.21.15.98:0/3050560752 conn(0x7fdca8005c00 msgr2=0x7fdca8034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:55:49.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:49.295+0000 7fdcaf59e640 1 -- 172.21.15.98:0/3050560752 shutdown_connections 2024-03-29T11:55:49.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:49.295+0000 7fdcaf59e640 1 -- 172.21.15.98:0/3050560752 wait complete. 2024-03-29T11:55:49.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:49.295+0000 7fdcaf59e640 1 librados: shutdown 2024-03-29T11:55:49.317 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:55:56.124 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:55:56.278 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.269+0000 7f568fcf4640 10 monclient: get_monmap_and_config 2024-03-29T11:55:56.278 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.269+0000 7f568fcf4640 10 monclient: build_initial_monmap 2024-03-29T11:55:56.278 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.269+0000 7f568fcf4640 10 monclient: monmap: 2024-03-29T11:55:56.278 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:55:56.278 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:55:56.278 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:55:56.270533+0000 2024-03-29T11:55:56.278 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:55:56.270533+0000 2024-03-29T11:55:56.278 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:55:56.278 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:56.278 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:55:56.279 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:55:56.279 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:55:56.279 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:56.280 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.272+0000 7f568fcf4640 1 Processor -- start 2024-03-29T11:55:56.282 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.272+0000 7f568fcf4640 1 -- start start 2024-03-29T11:55:56.282 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.272+0000 7f568fcf4640 10 monclient: init 2024-03-29T11:55:56.282 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.273+0000 7f568fcf4640 10 monclient: _reopen_session rank -1 2024-03-29T11:55:56.282 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.273+0000 7f568fcf4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:55:56.282 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.273+0000 7f568fcf4640 10 monclient(hunting): picked mon.noname-b con 0x7f56881457f0 addr v1:172.21.15.98:6790/0 2024-03-29T11:55:56.282 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.273+0000 7f568fcf4640 10 monclient(hunting): picked mon.noname-a con 0x7f568813f650 addr v1:172.21.15.98:6789/0 2024-03-29T11:55:56.282 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.273+0000 7f568fcf4640 10 monclient(hunting): picked mon.noname-c con 0x7f56880a9d70 addr v1:172.21.15.121:6789/0 2024-03-29T11:55:56.282 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.273+0000 7f568fcf4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56880aaef0 con 0x7f568813f650 2024-03-29T11:55:56.282 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.273+0000 7f568fcf4640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5688145bd0 con 0x7f56880a9d70 2024-03-29T11:55:56.282 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.273+0000 7f568fcf4640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5688145eb0 con 0x7f56881457f0 2024-03-29T11:55:56.283 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.273+0000 7f568fcf4640 10 monclient(hunting): _renew_subs 2024-03-29T11:55:56.283 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.273+0000 7f568fcf4640 10 monclient(hunting): authenticate will time out at 1263.358652s 2024-03-29T11:55:56.285 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.276+0000 7f568e291640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f56880a9d70 0x7f568814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:36341/0 (socket says 172.21.15.98:36341) 2024-03-29T11:55:56.285 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.277+0000 7f568e291640 1 -- 172.21.15.98:0/1190890577 learned_addr learned my addr 172.21.15.98:0/1190890577 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:55:56.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.281+0000 7f568ca8e640 1 -- 172.21.15.98:0/1190890577 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2499308476 0 0) 0x7f56880aaef0 con 0x7f568813f650 2024-03-29T11:55:56.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.281+0000 7f568ca8e640 10 monclient(hunting): _negotiate 2024-03-29T11:55:56.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.281+0000 7f568ca8e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:55:56.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.281+0000 7f568ca8e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:55:56.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.281+0000 7f568ca8e640 10 monclient(hunting): my global_id is 4482 2024-03-29T11:55:56.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.281+0000 7f568ca8e640 1 -- 172.21.15.98:0/1190890577 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5674003520 con 0x7f568813f650 2024-03-29T11:55:56.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.281+0000 7f568ca8e640 1 -- 172.21.15.98:0/1190890577 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 498704463 0 0) 0x7f5688145bd0 con 0x7f56880a9d70 2024-03-29T11:55:56.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.281+0000 7f568ca8e640 10 monclient(hunting): _negotiate 2024-03-29T11:55:56.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.281+0000 7f568ca8e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:55:56.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.281+0000 7f568ca8e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:55:56.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.281+0000 7f568ca8e640 10 monclient(hunting): my global_id is 4501 2024-03-29T11:55:56.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.281+0000 7f568ca8e640 1 -- 172.21.15.98:0/1190890577 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56880aaef0 con 0x7f56880a9d70 2024-03-29T11:55:56.291 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.282+0000 7f568ca8e640 1 -- 172.21.15.98:0/1190890577 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3921765338 0 0) 0x7f56880aaef0 con 0x7f56880a9d70 2024-03-29T11:55:56.291 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.282+0000 7f568ca8e640 1 -- 172.21.15.98:0/1190890577 >> v1:172.21.15.98:6790/0 conn(0x7f56881457f0 legacy=0x7f568813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:56.292 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.282+0000 7f568ca8e640 1 -- 172.21.15.98:0/1190890577 >> v1:172.21.15.98:6789/0 conn(0x7f568813f650 legacy=0x7f56880b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:56.292 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.282+0000 7f568ca8e640 10 monclient: _finish_hunting 0 2024-03-29T11:55:56.292 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.282+0000 7f568ca8e640 1 monclient: found mon.noname-c 2024-03-29T11:55:56.292 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.282+0000 7f568ca8e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:55:56.292 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.282+0000 7f568ca8e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:55:56.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.285+0000 7f568ca8e640 1 -- 172.21.15.98:0/1190890577 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5688146340 con 0x7f56880a9d70 2024-03-29T11:55:56.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.286+0000 7f568ca8e640 10 monclient: _finish_auth 0 2024-03-29T11:55:56.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.286+0000 7f568ca8e640 10 monclient: _check_auth_tickets 2024-03-29T11:55:56.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.286+0000 7f568ca8e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:55:56.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.286+0000 7f568fcf4640 5 monclient: authenticate success, global_id 4501 2024-03-29T11:55:56.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.286+0000 7f568fcf4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:55:56.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.286+0000 7f568ca8e640 1 -- 172.21.15.98:0/1190890577 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f5678001670 con 0x7f56880a9d70 2024-03-29T11:55:56.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.286+0000 7f568ca8e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:55:56.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.286+0000 7f568ca8e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:55:56.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.286+0000 7f568ca8e640 10 monclient: dump: 2024-03-29T11:55:56.295 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:55:56.295 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:55:56.295 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:56.295 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:56.296 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:55:56.296 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:56.296 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:55:56.296 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:55:56.296 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:55:56.296 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:56.296 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.286+0000 7f568fcf4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:55:56.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.287+0000 7f568ca8e640 1 -- 172.21.15.98:0/1190890577 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f56780017d0 con 0x7f56880a9d70 2024-03-29T11:55:56.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.287+0000 7f568ca8e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:55:56.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.287+0000 7f568ca8e640 1 -- 172.21.15.98:0/1190890577 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f5678001b60 con 0x7f56880a9d70 2024-03-29T11:55:56.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.287+0000 7f568ca8e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:55:56.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.287+0000 7f568ca8e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:55:56.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.287+0000 7f568ca8e640 10 monclient: dump: 2024-03-29T11:55:56.297 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:55:56.297 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:55:56.297 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:56.297 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:56.297 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:55:56.297 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:56.298 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:55:56.298 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:55:56.298 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:55:56.298 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:56.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.287+0000 7f568fcf4640 10 monclient: get_monmap_and_config success 2024-03-29T11:55:56.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.287+0000 7f568fcf4640 4 set_mon_vals no callback set 2024-03-29T11:55:56.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.288+0000 7f568fcf4640 10 monclient: shutdown 2024-03-29T11:55:56.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.288+0000 7f568fcf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:55:56.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.288+0000 7f568fcf4640 1 -- 172.21.15.98:0/1190890577 >> v1:172.21.15.121:6789/0 conn(0x7f56880a9d70 legacy=0x7f568814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:56.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.291+0000 7f568fcf4640 1 -- 172.21.15.98:0/1190890577 shutdown_connections 2024-03-29T11:55:56.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.291+0000 7f568fcf4640 1 -- 172.21.15.98:0/1190890577 >> 172.21.15.98:0/1190890577 conn(0x7f5688066a00 msgr2=0x7f5688066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:55:56.300 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.292+0000 7f568fcf4640 1 -- 172.21.15.98:0/1190890577 shutdown_connections 2024-03-29T11:55:56.300 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.292+0000 7f568fcf4640 1 -- 172.21.15.98:0/1190890577 wait complete. 2024-03-29T11:55:56.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.293+0000 7f568fcf4640 10 monclient: build_initial_monmap 2024-03-29T11:55:56.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.293+0000 7f568fcf4640 10 monclient: monmap: 2024-03-29T11:55:56.302 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:55:56.302 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:55:56.302 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:55:56.302 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:55:56.302 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:55:56.302 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:56.302 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:55:56.302 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:55:56.303 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:55:56.303 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:56.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.293+0000 7f568fcf4640 1 librados: starting msgr at 2024-03-29T11:55:56.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.293+0000 7f568fcf4640 1 librados: starting objecter 2024-03-29T11:55:56.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.293+0000 7f568fcf4640 1 Processor -- start 2024-03-29T11:55:56.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.297+0000 7f568fcf4640 1 -- start start 2024-03-29T11:55:56.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.297+0000 7f568fcf4640 1 librados: setting wanted keys 2024-03-29T11:55:56.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.297+0000 7f568fcf4640 1 librados: calling monclient init 2024-03-29T11:55:56.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.297+0000 7f568fcf4640 10 monclient: init 2024-03-29T11:55:56.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.298+0000 7f568fcf4640 10 monclient: _reopen_session rank -1 2024-03-29T11:55:56.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.298+0000 7f568fcf4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:55:56.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.298+0000 7f568fcf4640 10 monclient(hunting): picked mon.noname-a con 0x7f56880a9d70 addr v1:172.21.15.98:6789/0 2024-03-29T11:55:56.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.298+0000 7f568fcf4640 10 monclient(hunting): picked mon.noname-c con 0x7f568813f650 addr v1:172.21.15.98:6790/0 2024-03-29T11:55:56.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.298+0000 7f568fcf4640 10 monclient(hunting): picked mon.noname-b con 0x7f56881457f0 addr v1:172.21.15.121:6789/0 2024-03-29T11:55:56.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.298+0000 7f568fcf4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5688142150 con 0x7f56880a9d70 2024-03-29T11:55:56.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.298+0000 7f568fcf4640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5688142cf0 con 0x7f56881457f0 2024-03-29T11:55:56.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.298+0000 7f568fcf4640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5688141e10 con 0x7f568813f650 2024-03-29T11:55:56.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.298+0000 7f568fcf4640 10 monclient(hunting): _renew_subs 2024-03-29T11:55:56.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.298+0000 7f568fcf4640 10 monclient(hunting): authenticate will time out at 1263.383977s 2024-03-29T11:55:56.312 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.302+0000 7f568da90640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f56880a9d70 0x7f56881425e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56229/0 (socket says 172.21.15.98:56229) 2024-03-29T11:55:56.312 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.302+0000 7f568da90640 1 -- 172.21.15.98:0/665146271 learned_addr learned my addr 172.21.15.98:0/665146271 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:55:56.312 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.302+0000 7f567e7fc640 10 client.?.objecter ms_handle_connect 0x7f56880a9d70 2024-03-29T11:55:56.312 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.302+0000 7f567e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:55:56.312 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.303+0000 7f567e7fc640 1 -- 172.21.15.98:0/665146271 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3387739927 0 0) 0x7f5688142150 con 0x7f56880a9d70 2024-03-29T11:55:56.312 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.303+0000 7f567e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:55:56.312 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.303+0000 7f567e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:55:56.313 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.303+0000 7f567e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:55:56.313 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.303+0000 7f567e7fc640 10 monclient(hunting): my global_id is 4485 2024-03-29T11:55:56.313 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.303+0000 7f567e7fc640 1 -- 172.21.15.98:0/665146271 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f566c004eb0 con 0x7f56880a9d70 2024-03-29T11:55:56.313 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.303+0000 7f567e7fc640 1 -- 172.21.15.98:0/665146271 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3092934947 0 0) 0x7f566c004eb0 con 0x7f56880a9d70 2024-03-29T11:55:56.313 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.303+0000 7f567e7fc640 1 -- 172.21.15.98:0/665146271 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5688142150 con 0x7f56880a9d70 2024-03-29T11:55:56.313 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.303+0000 7f567e7fc640 1 -- 172.21.15.98:0/665146271 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f5680001cb0 con 0x7f56880a9d70 2024-03-29T11:55:56.313 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.303+0000 7f567e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:55:56.313 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.304+0000 7f567e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:55:56.313 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.304+0000 7f567e7fc640 10 monclient(hunting): dump: 2024-03-29T11:55:56.313 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:55:56.313 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:55:56.313 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:56.313 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:56.313 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:55:56.314 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:56.314 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:55:56.314 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:55:56.314 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:55:56.314 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:56.314 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.304+0000 7f567e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:55:56.314 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.304+0000 7f567e7fc640 1 -- 172.21.15.98:0/665146271 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1169181179 0 0) 0x7f5688142150 con 0x7f56880a9d70 2024-03-29T11:55:56.314 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.304+0000 7f567e7fc640 1 -- 172.21.15.98:0/665146271 >> v1:172.21.15.98:6790/0 conn(0x7f568813f650 legacy=0x7f5688141700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:55:56.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.307+0000 7f567e7fc640 1 -- 172.21.15.98:0/665146271 >> v1:172.21.15.121:6789/0 conn(0x7f56881457f0 legacy=0x7f56881e1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:56.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.307+0000 7f567e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:55:56.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.307+0000 7f567e7fc640 1 monclient: found mon.a 2024-03-29T11:55:56.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.307+0000 7f567e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:55:56.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.307+0000 7f567e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:55:56.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.307+0000 7f567e7fc640 1 -- 172.21.15.98:0/665146271 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56880ac390 con 0x7f56880a9d70 2024-03-29T11:55:56.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.307+0000 7f567e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:55:56.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.307+0000 7f567e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:55:56.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.307+0000 7f567e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:55:56.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.307+0000 7f568fcf4640 5 monclient: authenticate success, global_id 4485 2024-03-29T11:55:56.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.307+0000 7f568fcf4640 10 monclient: _renew_subs 2024-03-29T11:55:56.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.307+0000 7f568fcf4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:55:56.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.307+0000 7f568fcf4640 1 -- 172.21.15.98:0/665146271 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f568813fda0 con 0x7f56880a9d70 2024-03-29T11:55:56.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.307+0000 7f568fcf4640 10 client.4485.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:55:56.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.307+0000 7f568fcf4640 10 monclient: _renew_subs 2024-03-29T11:55:56.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.307+0000 7f568fcf4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:55:56.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.307+0000 7f568fcf4640 1 -- 172.21.15.98:0/665146271 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5688144980 con 0x7f56880a9d70 2024-03-29T11:55:56.318 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.307+0000 7f568fcf4640 1 librados: init done 2024-03-29T11:55:56.318 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.310+0000 7f568fcf4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:55:56.318 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.310+0000 7f568fcf4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:55:56.318 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.310+0000 7f568fcf4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:55:56.318 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.310+0000 7f568fcf4640 1 -- 172.21.15.98:0/665146271 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5688140140 con 0x7f56880a9d70 2024-03-29T11:55:56.320 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.311+0000 7f567e7fc640 1 -- 172.21.15.98:0/665146271 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f56800020e0 con 0x7f56880a9d70 2024-03-29T11:55:56.320 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.311+0000 7f567e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:55:56.320 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.312+0000 7f567e7fc640 1 -- 172.21.15.98:0/665146271 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f5680003de0 con 0x7f56880a9d70 2024-03-29T11:55:56.320 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.312+0000 7f567e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:55:56.320 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.312+0000 7f567e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:55:56.320 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.312+0000 7f567e7fc640 10 monclient: dump: 2024-03-29T11:55:56.320 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:55:56.321 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:55:56.321 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:56.321 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:55:56.321 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:55:56.321 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:55:56.321 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:55:56.321 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:55:56.321 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:55:56.321 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:55:56.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.312+0000 7f567e7fc640 1 -- 172.21.15.98:0/665146271 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f5680004000 con 0x7f56880a9d70 2024-03-29T11:55:56.322 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.312+0000 7f567f7fe640 4 set_mon_vals no callback set 2024-03-29T11:55:56.322 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.312+0000 7f567e7fc640 1 -- 172.21.15.98:0/665146271 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f5680084e90 con 0x7f56880a9d70 2024-03-29T11:55:56.322 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.313+0000 7f567e7fc640 10 client.4485.objecter ms_dispatch 0x7f5688000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:55:56.322 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.313+0000 7f567e7fc640 3 client.4485.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:55:56.322 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.313+0000 7f567e7fc640 3 client.4485.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:55:56.323 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.313+0000 7f567e7fc640 20 client.4485.objecter dump_active .. 0 homeless 2024-03-29T11:55:56.323 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.313+0000 7f568d28f640 10 monclient: get_auth_request con 0x7f566c06ddf0 auth_method 0 2024-03-29T11:55:56.323 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.313+0000 7f567e7fc640 10 client.4485.objecter ms_handle_connect 0x7f566c06ddf0 2024-03-29T11:55:56.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.321+0000 7f567e7fc640 1 -- 172.21.15.98:0/665146271 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f56800507f0 con 0x7f56880a9d70 2024-03-29T11:55:56.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.321+0000 7f567e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:55:56.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.321+0000 7f567e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:55:56.674 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.665+0000 7f568fcf4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:55:56.674 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.665+0000 7f568fcf4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:55:56.674 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.665+0000 7f568fcf4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:55:56.674 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.665+0000 7f568fcf4640 1 -- 172.21.15.98:0/665146271 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56880ac1d0 con 0x7f56880a9d70 2024-03-29T11:55:56.675 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.666+0000 7f567e7fc640 1 -- 172.21.15.98:0/665146271 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f5680052720 con 0x7f56880a9d70 2024-03-29T11:55:56.675 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.666+0000 7f567e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:55:56.675 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.666+0000 7f567e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:55:56.675 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:55:56.675 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:55:56.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.670+0000 7f568fcf4640 10 librados: watch_flush enter 2024-03-29T11:55:56.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.670+0000 7f568fcf4640 10 librados: watch_flush exit 2024-03-29T11:55:56.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.670+0000 7f568fcf4640 20 client.4485.objecter shutdown clearing up homeless session... 2024-03-29T11:55:56.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.670+0000 7f568fcf4640 10 client.4485.objecter successfully canceled tick 2024-03-29T11:55:56.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.670+0000 7f568fcf4640 1 -- 172.21.15.98:0/665146271 >> v1:172.21.15.98:6816/4065186974 conn(0x7f566c06ddf0 legacy=0x7f566c0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:56.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.670+0000 7f568fcf4640 10 monclient: shutdown 2024-03-29T11:55:56.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.670+0000 7f568fcf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:55:56.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.670+0000 7f568fcf4640 1 -- 172.21.15.98:0/665146271 >> v1:172.21.15.98:6789/0 conn(0x7f56880a9d70 legacy=0x7f56881425e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:55:56.680 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.671+0000 7f568fcf4640 1 -- 172.21.15.98:0/665146271 shutdown_connections 2024-03-29T11:55:56.680 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.671+0000 7f568fcf4640 1 -- 172.21.15.98:0/665146271 >> 172.21.15.98:0/665146271 conn(0x7f5688066a00 msgr2=0x7f5688066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:55:56.680 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.671+0000 7f568fcf4640 1 -- 172.21.15.98:0/665146271 shutdown_connections 2024-03-29T11:55:56.680 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.671+0000 7f568fcf4640 1 -- 172.21.15.98:0/665146271 wait complete. 2024-03-29T11:55:56.680 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:55:56.671+0000 7f568fcf4640 1 librados: shutdown 2024-03-29T11:55:56.693 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:56:03.501 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:56:03.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.627+0000 7f8b26cd9640 10 monclient: get_monmap_and_config 2024-03-29T11:56:03.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.627+0000 7f8b26cd9640 10 monclient: build_initial_monmap 2024-03-29T11:56:03.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.627+0000 7f8b26cd9640 10 monclient: monmap: 2024-03-29T11:56:03.636 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:56:03.636 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:56:03.636 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:56:03.628013+0000 2024-03-29T11:56:03.636 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:56:03.628013+0000 2024-03-29T11:56:03.636 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:56:03.636 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:03.636 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:56:03.636 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:56:03.636 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:56:03.636 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:03.638 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.628+0000 7f8b26cd9640 1 Processor -- start 2024-03-29T11:56:03.638 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.628+0000 7f8b26cd9640 1 -- start start 2024-03-29T11:56:03.638 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.628+0000 7f8b26cd9640 10 monclient: init 2024-03-29T11:56:03.638 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.629+0000 7f8b26cd9640 10 monclient: _reopen_session rank -1 2024-03-29T11:56:03.638 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.629+0000 7f8b26cd9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:56:03.638 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.629+0000 7f8b26cd9640 10 monclient(hunting): picked mon.noname-a con 0x7f8b201457f0 addr v1:172.21.15.98:6789/0 2024-03-29T11:56:03.638 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.629+0000 7f8b26cd9640 10 monclient(hunting): picked mon.noname-b con 0x7f8b2013f650 addr v1:172.21.15.98:6790/0 2024-03-29T11:56:03.638 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.629+0000 7f8b26cd9640 10 monclient(hunting): picked mon.noname-c con 0x7f8b200a9d70 addr v1:172.21.15.121:6789/0 2024-03-29T11:56:03.638 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.629+0000 7f8b26cd9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b200aaef0 con 0x7f8b201457f0 2024-03-29T11:56:03.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.629+0000 7f8b26cd9640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b20145bd0 con 0x7f8b200a9d70 2024-03-29T11:56:03.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.629+0000 7f8b26cd9640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b20145eb0 con 0x7f8b2013f650 2024-03-29T11:56:03.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.629+0000 7f8b26cd9640 10 monclient(hunting): _renew_subs 2024-03-29T11:56:03.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.629+0000 7f8b26cd9640 10 monclient(hunting): authenticate will time out at 1270.714284s 2024-03-29T11:56:03.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.629+0000 7f8b24a75640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f8b201457f0 0x7f8b2013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43044/0 (socket says 172.21.15.98:43044) 2024-03-29T11:56:03.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.629+0000 7f8b24a75640 1 -- 172.21.15.98:0/4010486727 learned_addr learned my addr 172.21.15.98:0/4010486727 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:56:03.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.629+0000 7f8b1f7fe640 1 -- 172.21.15.98:0/4010486727 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1239632674 0 0) 0x7f8b20145eb0 con 0x7f8b2013f650 2024-03-29T11:56:03.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.630+0000 7f8b1f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:56:03.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.630+0000 7f8b1f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:56:03.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.630+0000 7f8b1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:56:03.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.630+0000 7f8b1f7fe640 10 monclient(hunting): my global_id is 4523 2024-03-29T11:56:03.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.630+0000 7f8b1f7fe640 1 -- 172.21.15.98:0/4010486727 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b10003520 con 0x7f8b2013f650 2024-03-29T11:56:03.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.630+0000 7f8b1f7fe640 1 -- 172.21.15.98:0/4010486727 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1022503218 0 0) 0x7f8b10003520 con 0x7f8b2013f650 2024-03-29T11:56:03.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.630+0000 7f8b1f7fe640 1 -- 172.21.15.98:0/4010486727 >> v1:172.21.15.121:6789/0 conn(0x7f8b200a9d70 legacy=0x7f8b2014d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:56:03.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.631+0000 7f8b1f7fe640 1 -- 172.21.15.98:0/4010486727 >> v1:172.21.15.98:6789/0 conn(0x7f8b201457f0 legacy=0x7f8b2013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:03.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.631+0000 7f8b1f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:56:03.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.631+0000 7f8b1f7fe640 1 monclient: found mon.noname-b 2024-03-29T11:56:03.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.631+0000 7f8b1f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:56:03.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.631+0000 7f8b1f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:56:03.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.631+0000 7f8b1f7fe640 1 -- 172.21.15.98:0/4010486727 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b20146340 con 0x7f8b2013f650 2024-03-29T11:56:03.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.631+0000 7f8b1f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:56:03.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.631+0000 7f8b1f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:56:03.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.631+0000 7f8b1f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:56:03.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.631+0000 7f8b1f7fe640 1 -- 172.21.15.98:0/4010486727 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f8b0c001690 con 0x7f8b2013f650 2024-03-29T11:56:03.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.631+0000 7f8b1f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:56:03.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.631+0000 7f8b1f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:56:03.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.631+0000 7f8b1f7fe640 10 monclient: dump: 2024-03-29T11:56:03.642 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:56:03.642 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:56:03.642 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:03.642 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:03.642 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:56:03.642 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:03.643 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:56:03.643 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:56:03.643 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:56:03.643 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:03.643 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.631+0000 7f8b26cd9640 5 monclient: authenticate success, global_id 4523 2024-03-29T11:56:03.643 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.631+0000 7f8b26cd9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:56:03.643 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.633+0000 7f8b1f7fe640 1 -- 172.21.15.98:0/4010486727 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b0c001920 con 0x7f8b2013f650 2024-03-29T11:56:03.643 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.633+0000 7f8b1f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:56:03.643 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.633+0000 7f8b26cd9640 10 monclient: get_monmap_and_config success 2024-03-29T11:56:03.643 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.633+0000 7f8b26cd9640 4 set_mon_vals no callback set 2024-03-29T11:56:03.643 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.633+0000 7f8b1f7fe640 1 -- 172.21.15.98:0/4010486727 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f8b0c001cb0 con 0x7f8b2013f650 2024-03-29T11:56:03.643 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.633+0000 7f8b1f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:56:03.643 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.633+0000 7f8b1f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:56:03.643 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.633+0000 7f8b1f7fe640 10 monclient: dump: 2024-03-29T11:56:03.644 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:56:03.644 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:56:03.644 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:03.644 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:03.644 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:56:03.644 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:03.644 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:56:03.644 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:56:03.644 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:56:03.644 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:03.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.634+0000 7f8b26cd9640 10 monclient: shutdown 2024-03-29T11:56:03.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.634+0000 7f8b26cd9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:56:03.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.634+0000 7f8b26cd9640 1 -- 172.21.15.98:0/4010486727 >> v1:172.21.15.98:6790/0 conn(0x7f8b2013f650 legacy=0x7f8b200b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:03.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.634+0000 7f8b26cd9640 1 -- 172.21.15.98:0/4010486727 shutdown_connections 2024-03-29T11:56:03.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.634+0000 7f8b26cd9640 1 -- 172.21.15.98:0/4010486727 >> 172.21.15.98:0/4010486727 conn(0x7f8b20066a00 msgr2=0x7f8b20066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:56:03.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.634+0000 7f8b26cd9640 1 -- 172.21.15.98:0/4010486727 shutdown_connections 2024-03-29T11:56:03.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.634+0000 7f8b26cd9640 1 -- 172.21.15.98:0/4010486727 wait complete. 2024-03-29T11:56:03.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.635+0000 7f8b26cd9640 10 monclient: build_initial_monmap 2024-03-29T11:56:03.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.635+0000 7f8b26cd9640 10 monclient: monmap: 2024-03-29T11:56:03.652 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:56:03.652 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:56:03.652 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:56:03.652 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:56:03.652 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:56:03.652 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:03.653 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:56:03.653 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:56:03.653 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:56:03.653 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:03.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.635+0000 7f8b26cd9640 1 librados: starting msgr at 2024-03-29T11:56:03.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.635+0000 7f8b26cd9640 1 librados: starting objecter 2024-03-29T11:56:03.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.635+0000 7f8b26cd9640 1 Processor -- start 2024-03-29T11:56:03.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.635+0000 7f8b26cd9640 1 -- start start 2024-03-29T11:56:03.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.635+0000 7f8b26cd9640 1 librados: setting wanted keys 2024-03-29T11:56:03.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.635+0000 7f8b26cd9640 1 librados: calling monclient init 2024-03-29T11:56:03.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.635+0000 7f8b26cd9640 10 monclient: init 2024-03-29T11:56:03.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.636+0000 7f8b26cd9640 10 monclient: _reopen_session rank -1 2024-03-29T11:56:03.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.636+0000 7f8b26cd9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:56:03.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.636+0000 7f8b26cd9640 10 monclient(hunting): picked mon.noname-c con 0x7f8b200a9d70 addr v1:172.21.15.98:6790/0 2024-03-29T11:56:03.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.636+0000 7f8b26cd9640 10 monclient(hunting): picked mon.noname-b con 0x7f8b2013f650 addr v1:172.21.15.121:6789/0 2024-03-29T11:56:03.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.636+0000 7f8b26cd9640 10 monclient(hunting): picked mon.noname-a con 0x7f8b201457f0 addr v1:172.21.15.98:6789/0 2024-03-29T11:56:03.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.636+0000 7f8b26cd9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b20070cf0 con 0x7f8b201457f0 2024-03-29T11:56:03.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.636+0000 7f8b26cd9640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b20070f60 con 0x7f8b2013f650 2024-03-29T11:56:03.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.636+0000 7f8b26cd9640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b201dbc70 con 0x7f8b200a9d70 2024-03-29T11:56:03.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.636+0000 7f8b26cd9640 10 monclient(hunting): _renew_subs 2024-03-29T11:56:03.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.636+0000 7f8b26cd9640 10 monclient(hunting): authenticate will time out at 1270.721853s 2024-03-29T11:56:03.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.637+0000 7f8b25276640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f8b201457f0 0x7f8b201e6f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43450/0 (socket says 172.21.15.98:43450) 2024-03-29T11:56:03.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.637+0000 7f8b25276640 1 -- 172.21.15.98:0/3058132920 learned_addr learned my addr 172.21.15.98:0/3058132920 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:56:03.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.637+0000 7f8b1d7fa640 10 client.?.objecter ms_handle_connect 0x7f8b201457f0 2024-03-29T11:56:03.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.637+0000 7f8b1d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:56:03.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.637+0000 7f8b1d7fa640 10 client.?.objecter ms_handle_connect 0x7f8b200a9d70 2024-03-29T11:56:03.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.637+0000 7f8b1d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:56:03.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.637+0000 7f8b1d7fa640 1 -- 172.21.15.98:0/3058132920 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2925060132 0 0) 0x7f8b20070cf0 con 0x7f8b201457f0 2024-03-29T11:56:03.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.637+0000 7f8b1d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:56:03.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.637+0000 7f8b1d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:56:03.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.638+0000 7f8b1d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:56:03.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.638+0000 7f8b1d7fa640 10 monclient(hunting): my global_id is 4491 2024-03-29T11:56:03.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.638+0000 7f8b1d7fa640 1 -- 172.21.15.98:0/3058132920 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b04004eb0 con 0x7f8b201457f0 2024-03-29T11:56:03.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.638+0000 7f8b1d7fa640 1 -- 172.21.15.98:0/3058132920 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 559680839 0 0) 0x7f8b201dbc70 con 0x7f8b200a9d70 2024-03-29T11:56:03.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.638+0000 7f8b1d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:56:03.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.638+0000 7f8b1d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:56:03.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.638+0000 7f8b1d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:56:03.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.638+0000 7f8b1d7fa640 10 monclient(hunting): my global_id is 4526 2024-03-29T11:56:03.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.638+0000 7f8b1d7fa640 1 -- 172.21.15.98:0/3058132920 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b20070cf0 con 0x7f8b200a9d70 2024-03-29T11:56:03.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.638+0000 7f8b1d7fa640 1 -- 172.21.15.98:0/3058132920 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1347287345 0 0) 0x7f8b20070cf0 con 0x7f8b200a9d70 2024-03-29T11:56:03.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.638+0000 7f8b1d7fa640 1 -- 172.21.15.98:0/3058132920 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b201dbc70 con 0x7f8b200a9d70 2024-03-29T11:56:03.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.638+0000 7f8b1d7fa640 1 -- 172.21.15.98:0/3058132920 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f8b14001790 con 0x7f8b200a9d70 2024-03-29T11:56:03.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.638+0000 7f8b1d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:56:03.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.638+0000 7f8b1d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:56:03.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.638+0000 7f8b1d7fa640 10 monclient(hunting): dump: 2024-03-29T11:56:03.656 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:56:03.656 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:56:03.656 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:03.656 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:03.656 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:56:03.656 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:03.656 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:56:03.656 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:56:03.656 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:56:03.656 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:03.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.639+0000 7f8b1d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:56:03.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.639+0000 7f8b1d7fa640 1 -- 172.21.15.98:0/3058132920 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1479439382 0 0) 0x7f8b04004eb0 con 0x7f8b201457f0 2024-03-29T11:56:03.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.639+0000 7f8b1d7fa640 1 -- 172.21.15.98:0/3058132920 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b20070cf0 con 0x7f8b201457f0 2024-03-29T11:56:03.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.639+0000 7f8b1d7fa640 1 -- 172.21.15.98:0/3058132920 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f8b18001b70 con 0x7f8b201457f0 2024-03-29T11:56:03.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.639+0000 7f8b1d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:56:03.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.639+0000 7f8b1d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:56:03.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.639+0000 7f8b1d7fa640 10 monclient(hunting): dump: 2024-03-29T11:56:03.657 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:56:03.657 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:56:03.657 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:03.657 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:03.657 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:56:03.657 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:03.657 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:56:03.657 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:56:03.657 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:56:03.657 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:03.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.639+0000 7f8b1d7fa640 1 -- 172.21.15.98:0/3058132920 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1082341623 0 0) 0x7f8b201dbc70 con 0x7f8b200a9d70 2024-03-29T11:56:03.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.639+0000 7f8b1d7fa640 1 -- 172.21.15.98:0/3058132920 >> v1:172.21.15.121:6789/0 conn(0x7f8b2013f650 legacy=0x7f8b2006f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:03.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.639+0000 7f8b1d7fa640 1 -- 172.21.15.98:0/3058132920 >> v1:172.21.15.98:6789/0 conn(0x7f8b201457f0 legacy=0x7f8b201e6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:03.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.639+0000 7f8b1d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:56:03.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.639+0000 7f8b1d7fa640 1 monclient: found mon.c 2024-03-29T11:56:03.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.639+0000 7f8b1d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:56:03.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.639+0000 7f8b1d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:56:03.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.639+0000 7f8b1d7fa640 1 -- 172.21.15.98:0/3058132920 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b200b1670 con 0x7f8b200a9d70 2024-03-29T11:56:03.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.639+0000 7f8b1d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:56:03.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.639+0000 7f8b1d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:56:03.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.639+0000 7f8b1d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:56:03.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.639+0000 7f8b26cd9640 5 monclient: authenticate success, global_id 4526 2024-03-29T11:56:03.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.639+0000 7f8b26cd9640 10 monclient: _renew_subs 2024-03-29T11:56:03.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.639+0000 7f8b26cd9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:56:03.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.639+0000 7f8b26cd9640 1 -- 172.21.15.98:0/3058132920 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b201db6c0 con 0x7f8b200a9d70 2024-03-29T11:56:03.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.639+0000 7f8b26cd9640 10 client.4526.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:56:03.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.639+0000 7f8b26cd9640 10 monclient: _renew_subs 2024-03-29T11:56:03.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.639+0000 7f8b26cd9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:56:03.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.639+0000 7f8b26cd9640 1 -- 172.21.15.98:0/3058132920 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b20071bb0 con 0x7f8b200a9d70 2024-03-29T11:56:03.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.639+0000 7f8b26cd9640 1 librados: init done 2024-03-29T11:56:03.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.641+0000 7f8b1d7fa640 1 -- 172.21.15.98:0/3058132920 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b14001d60 con 0x7f8b200a9d70 2024-03-29T11:56:03.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.641+0000 7f8b1d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:56:03.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.641+0000 7f8b26cd9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:56:03.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.641+0000 7f8b1d7fa640 1 -- 172.21.15.98:0/3058132920 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f8b14004980 con 0x7f8b200a9d70 2024-03-29T11:56:03.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.641+0000 7f8b26cd9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:56:03.660 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.641+0000 7f8b26cd9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:56:03.660 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.641+0000 7f8b26cd9640 1 -- 172.21.15.98:0/3058132920 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b201dbe00 con 0x7f8b200a9d70 2024-03-29T11:56:03.660 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.641+0000 7f8b1d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:56:03.660 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.641+0000 7f8b1d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:56:03.660 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.641+0000 7f8b1d7fa640 10 monclient: dump: 2024-03-29T11:56:03.660 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:56:03.660 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:56:03.660 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:03.660 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:03.660 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:56:03.660 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:03.660 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:56:03.661 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:56:03.661 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:56:03.661 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:03.661 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.641+0000 7f8b1effd640 4 set_mon_vals no callback set 2024-03-29T11:56:03.661 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.643+0000 7f8b1d7fa640 1 -- 172.21.15.98:0/3058132920 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f8b1401a950 con 0x7f8b200a9d70 2024-03-29T11:56:03.662 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.644+0000 7f8b1d7fa640 1 -- 172.21.15.98:0/3058132920 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f8b140848b0 con 0x7f8b200a9d70 2024-03-29T11:56:03.662 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.644+0000 7f8b1d7fa640 10 client.4526.objecter ms_dispatch 0x7f8b20000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:56:03.662 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.644+0000 7f8b1d7fa640 3 client.4526.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:56:03.662 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.644+0000 7f8b1d7fa640 3 client.4526.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:56:03.663 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.644+0000 7f8b1ffff640 10 monclient: get_auth_request con 0x7f8b0406de40 auth_method 0 2024-03-29T11:56:03.663 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.644+0000 7f8b1d7fa640 20 client.4526.objecter dump_active .. 0 homeless 2024-03-29T11:56:03.663 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.645+0000 7f8b1d7fa640 10 client.4526.objecter ms_handle_connect 0x7f8b0406de40 2024-03-29T11:56:03.663 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.648+0000 7f8b1d7fa640 1 -- 172.21.15.98:0/3058132920 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8b1404ffc0 con 0x7f8b200a9d70 2024-03-29T11:56:03.663 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.648+0000 7f8b1d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:56:03.663 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.648+0000 7f8b1d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:56:03.970 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.961+0000 7f8b26cd9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:56:03.970 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.961+0000 7f8b26cd9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:56:03.970 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.961+0000 7f8b26cd9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:56:03.970 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.961+0000 7f8b26cd9640 1 -- 172.21.15.98:0/3058132920 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b201dba60 con 0x7f8b200a9d70 2024-03-29T11:56:03.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.963+0000 7f8b1d7fa640 1 -- 172.21.15.98:0/3058132920 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f8b140520c0 con 0x7f8b200a9d70 2024-03-29T11:56:03.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.963+0000 7f8b1d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:56:03.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.963+0000 7f8b1d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:56:03.973 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:56:03.973 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:56:03.976 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.968+0000 7f8b26cd9640 10 librados: watch_flush enter 2024-03-29T11:56:03.977 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.968+0000 7f8b26cd9640 10 librados: watch_flush exit 2024-03-29T11:56:03.977 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.968+0000 7f8b26cd9640 20 client.4526.objecter shutdown clearing up homeless session... 2024-03-29T11:56:03.977 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.968+0000 7f8b26cd9640 10 client.4526.objecter successfully canceled tick 2024-03-29T11:56:03.977 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.968+0000 7f8b26cd9640 1 -- 172.21.15.98:0/3058132920 >> v1:172.21.15.98:6816/4065186974 conn(0x7f8b0406de40 legacy=0x7f8b04070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:03.977 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.968+0000 7f8b26cd9640 10 monclient: shutdown 2024-03-29T11:56:03.977 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.968+0000 7f8b26cd9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:56:03.977 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.968+0000 7f8b26cd9640 1 -- 172.21.15.98:0/3058132920 >> v1:172.21.15.98:6790/0 conn(0x7f8b200a9d70 legacy=0x7f8b2006eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:03.977 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.969+0000 7f8b26cd9640 1 -- 172.21.15.98:0/3058132920 shutdown_connections 2024-03-29T11:56:03.977 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.969+0000 7f8b26cd9640 1 -- 172.21.15.98:0/3058132920 >> 172.21.15.98:0/3058132920 conn(0x7f8b20066a00 msgr2=0x7f8b20066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:56:03.978 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.969+0000 7f8b26cd9640 1 -- 172.21.15.98:0/3058132920 shutdown_connections 2024-03-29T11:56:03.978 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.969+0000 7f8b26cd9640 1 -- 172.21.15.98:0/3058132920 wait complete. 2024-03-29T11:56:03.978 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:03.969+0000 7f8b26cd9640 1 librados: shutdown 2024-03-29T11:56:03.995 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:56:10.801 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:56:10.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.942+0000 7f986de49640 10 monclient: get_monmap_and_config 2024-03-29T11:56:10.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.942+0000 7f986de49640 10 monclient: build_initial_monmap 2024-03-29T11:56:10.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.942+0000 7f986de49640 10 monclient: monmap: 2024-03-29T11:56:10.951 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:56:10.951 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:56:10.951 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:56:10.943599+0000 2024-03-29T11:56:10.951 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:56:10.943599+0000 2024-03-29T11:56:10.951 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:56:10.952 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:10.952 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:56:10.952 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:56:10.952 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:56:10.952 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:10.952 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.944+0000 7f986de49640 1 Processor -- start 2024-03-29T11:56:10.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.944+0000 7f986de49640 1 -- start start 2024-03-29T11:56:10.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.944+0000 7f986de49640 10 monclient: init 2024-03-29T11:56:10.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.945+0000 7f986de49640 10 monclient: _reopen_session rank -1 2024-03-29T11:56:10.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.945+0000 7f986de49640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:56:10.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.945+0000 7f986de49640 10 monclient(hunting): picked mon.noname-b con 0x7f98680b0770 addr v1:172.21.15.98:6790/0 2024-03-29T11:56:10.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.945+0000 7f986de49640 10 monclient(hunting): picked mon.noname-a con 0x7f986814a700 addr v1:172.21.15.98:6789/0 2024-03-29T11:56:10.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.945+0000 7f986de49640 10 monclient(hunting): picked mon.noname-c con 0x7f98680ac5a0 addr v1:172.21.15.121:6789/0 2024-03-29T11:56:10.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.945+0000 7f986de49640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98680aba70 con 0x7f986814a700 2024-03-29T11:56:10.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.945+0000 7f986de49640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98680aecb0 con 0x7f98680ac5a0 2024-03-29T11:56:10.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.945+0000 7f986de49640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98680af150 con 0x7f98680b0770 2024-03-29T11:56:10.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.945+0000 7f986de49640 10 monclient(hunting): _renew_subs 2024-03-29T11:56:10.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.945+0000 7f986de49640 10 monclient(hunting): authenticate will time out at 1278.030828s 2024-03-29T11:56:10.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.946+0000 7f9866ffd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f986814a700 0x7f98680abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43564/0 (socket says 172.21.15.98:43564) 2024-03-29T11:56:10.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.946+0000 7f9866ffd640 1 -- 172.21.15.98:0/3073396224 learned_addr learned my addr 172.21.15.98:0/3073396224 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:56:10.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.946+0000 7f98667fc640 1 -- 172.21.15.98:0/3073396224 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1476574402 0 0) 0x7f98680aba70 con 0x7f986814a700 2024-03-29T11:56:10.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.946+0000 7f98667fc640 10 monclient(hunting): _negotiate 2024-03-29T11:56:10.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.946+0000 7f98667fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:56:10.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.946+0000 7f98667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:56:10.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.946+0000 7f98667fc640 10 monclient(hunting): my global_id is 4494 2024-03-29T11:56:10.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.946+0000 7f98667fc640 1 -- 172.21.15.98:0/3073396224 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9850014530 con 0x7f986814a700 2024-03-29T11:56:10.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.946+0000 7f98667fc640 1 -- 172.21.15.98:0/3073396224 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3942984248 0 0) 0x7f9850014530 con 0x7f986814a700 2024-03-29T11:56:10.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.946+0000 7f98667fc640 1 -- 172.21.15.98:0/3073396224 >> v1:172.21.15.98:6790/0 conn(0x7f98680b0770 legacy=0x7f9868149ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:56:10.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.946+0000 7f98667fc640 1 -- 172.21.15.98:0/3073396224 >> v1:172.21.15.121:6789/0 conn(0x7f98680ac5a0 legacy=0x7f9868151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:56:10.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.947+0000 7f98667fc640 10 monclient: _finish_hunting 0 2024-03-29T11:56:10.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.947+0000 7f98667fc640 1 monclient: found mon.noname-a 2024-03-29T11:56:10.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.947+0000 7f98667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:56:10.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.947+0000 7f98667fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T11:56:10.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.947+0000 7f98667fc640 1 -- 172.21.15.98:0/3073396224 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98680ac9d0 con 0x7f986814a700 2024-03-29T11:56:10.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.947+0000 7f98667fc640 10 monclient: _finish_auth 0 2024-03-29T11:56:10.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.947+0000 7f98667fc640 10 monclient: _check_auth_tickets 2024-03-29T11:56:10.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.947+0000 7f98667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:56:10.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.947+0000 7f98667fc640 1 -- 172.21.15.98:0/3073396224 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f985c001670 con 0x7f986814a700 2024-03-29T11:56:10.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.947+0000 7f98667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:56:10.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.947+0000 7f98667fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:56:10.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.947+0000 7f98667fc640 10 monclient: dump: 2024-03-29T11:56:10.960 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:56:10.960 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:56:10.960 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:10.960 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:10.960 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:56:10.960 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:10.960 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:56:10.960 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:56:10.960 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:56:10.960 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:10.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.947+0000 7f986de49640 5 monclient: authenticate success, global_id 4494 2024-03-29T11:56:10.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.947+0000 7f986de49640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:56:10.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.948+0000 7f98667fc640 1 -- 172.21.15.98:0/3073396224 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f985c0018d0 con 0x7f986814a700 2024-03-29T11:56:10.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.948+0000 7f98667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:56:10.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.948+0000 7f986de49640 10 monclient: get_monmap_and_config success 2024-03-29T11:56:10.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.948+0000 7f986de49640 4 set_mon_vals no callback set 2024-03-29T11:56:10.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.949+0000 7f986de49640 10 monclient: shutdown 2024-03-29T11:56:10.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.949+0000 7f986de49640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:56:10.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.949+0000 7f986de49640 1 -- 172.21.15.98:0/3073396224 >> v1:172.21.15.98:6789/0 conn(0x7f986814a700 legacy=0x7f98680abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:10.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.949+0000 7f98667fc640 1 -- 172.21.15.98:0/3073396224 <== mon.0 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f985c001d20 con 0x7f986814a700 2024-03-29T11:56:10.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.949+0000 7f98667fc640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:56:10.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.949+0000 7f986de49640 1 -- 172.21.15.98:0/3073396224 shutdown_connections 2024-03-29T11:56:10.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.949+0000 7f986de49640 1 -- 172.21.15.98:0/3073396224 >> 172.21.15.98:0/3073396224 conn(0x7f986806b660 msgr2=0x7f986806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:56:10.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.950+0000 7f986de49640 1 -- 172.21.15.98:0/3073396224 shutdown_connections 2024-03-29T11:56:10.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.950+0000 7f986de49640 1 -- 172.21.15.98:0/3073396224 wait complete. 2024-03-29T11:56:10.963 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.951+0000 7f986de49640 10 monclient: build_initial_monmap 2024-03-29T11:56:10.963 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.951+0000 7f986de49640 10 monclient: monmap: 2024-03-29T11:56:10.963 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:56:10.963 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:56:10.963 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:56:10.963 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:56:10.963 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:56:10.963 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:10.963 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:56:10.963 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:56:10.963 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:56:10.964 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:10.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.951+0000 7f986de49640 1 librados: starting msgr at 2024-03-29T11:56:10.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.951+0000 7f986de49640 1 librados: starting objecter 2024-03-29T11:56:10.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.952+0000 7f986de49640 1 Processor -- start 2024-03-29T11:56:10.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.952+0000 7f986de49640 1 -- start start 2024-03-29T11:56:10.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.952+0000 7f986de49640 1 librados: setting wanted keys 2024-03-29T11:56:10.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.952+0000 7f986de49640 1 librados: calling monclient init 2024-03-29T11:56:10.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.952+0000 7f986de49640 10 monclient: init 2024-03-29T11:56:10.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.953+0000 7f986de49640 10 monclient: _reopen_session rank -1 2024-03-29T11:56:10.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.953+0000 7f986de49640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:56:10.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.954+0000 7f986de49640 10 monclient(hunting): picked mon.noname-c con 0x7f98680ac5a0 addr v1:172.21.15.98:6790/0 2024-03-29T11:56:10.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.954+0000 7f986de49640 10 monclient(hunting): picked mon.noname-a con 0x7f98680b0770 addr v1:172.21.15.98:6789/0 2024-03-29T11:56:10.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.954+0000 7f986de49640 10 monclient(hunting): picked mon.noname-b con 0x7f986814a700 addr v1:172.21.15.121:6789/0 2024-03-29T11:56:10.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.954+0000 7f986de49640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98681d77d0 con 0x7f98680b0770 2024-03-29T11:56:10.965 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.954+0000 7f986de49640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f986813f0f0 con 0x7f986814a700 2024-03-29T11:56:10.965 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.954+0000 7f986de49640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f986813f3d0 con 0x7f98680ac5a0 2024-03-29T11:56:10.965 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.954+0000 7f986de49640 10 monclient(hunting): _renew_subs 2024-03-29T11:56:10.965 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.954+0000 7f986de49640 10 monclient(hunting): authenticate will time out at 1278.039791s 2024-03-29T11:56:10.967 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.955+0000 7f9866ffd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f98680b0770 0x7f98681d69b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44120/0 (socket says 172.21.15.98:44120) 2024-03-29T11:56:10.967 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.955+0000 7f9866ffd640 1 -- 172.21.15.98:0/1577195068 learned_addr learned my addr 172.21.15.98:0/1577195068 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:56:10.967 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.955+0000 7f984ffff640 10 client.?.objecter ms_handle_connect 0x7f98680b0770 2024-03-29T11:56:10.968 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.955+0000 7f984ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:56:10.968 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.955+0000 7f984ffff640 1 -- 172.21.15.98:0/1577195068 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2415904031 0 0) 0x7f98681d77d0 con 0x7f98680b0770 2024-03-29T11:56:10.968 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.955+0000 7f984ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:56:10.968 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.955+0000 7f984ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:56:10.968 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.955+0000 7f984ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:56:10.968 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.955+0000 7f984ffff640 10 monclient(hunting): my global_id is 4497 2024-03-29T11:56:10.968 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.955+0000 7f984ffff640 1 -- 172.21.15.98:0/1577195068 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9854004eb0 con 0x7f98680b0770 2024-03-29T11:56:10.968 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.956+0000 7f984ffff640 1 -- 172.21.15.98:0/1577195068 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3662354467 0 0) 0x7f9854004eb0 con 0x7f98680b0770 2024-03-29T11:56:10.968 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.956+0000 7f984ffff640 1 -- 172.21.15.98:0/1577195068 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98681d77d0 con 0x7f98680b0770 2024-03-29T11:56:10.968 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.956+0000 7f984ffff640 1 -- 172.21.15.98:0/1577195068 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f985c001630 con 0x7f98680b0770 2024-03-29T11:56:10.968 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.956+0000 7f984ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:56:10.968 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.956+0000 7f984ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:56:10.968 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.956+0000 7f984ffff640 10 monclient(hunting): dump: 2024-03-29T11:56:10.968 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:56:10.969 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:56:10.969 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:10.969 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:10.969 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:56:10.969 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:10.969 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:56:10.969 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:56:10.969 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:56:10.969 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:10.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.956+0000 7f984ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:56:10.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.957+0000 7f984ffff640 1 -- 172.21.15.98:0/1577195068 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2723954645 0 0) 0x7f98681d77d0 con 0x7f98680b0770 2024-03-29T11:56:10.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.957+0000 7f984ffff640 1 -- 172.21.15.98:0/1577195068 >> v1:172.21.15.98:6790/0 conn(0x7f98680ac5a0 legacy=0x7f986813d7e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:56:10.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.957+0000 7f984ffff640 1 -- 172.21.15.98:0/1577195068 >> v1:172.21.15.121:6789/0 conn(0x7f986814a700 legacy=0x7f98681d70c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:56:10.970 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.958+0000 7f984ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:56:10.970 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.958+0000 7f984ffff640 1 monclient: found mon.a 2024-03-29T11:56:10.970 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.958+0000 7f984ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:56:10.970 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.958+0000 7f984ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:56:10.970 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.958+0000 7f984ffff640 1 -- 172.21.15.98:0/1577195068 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f986813f6b0 con 0x7f98680b0770 2024-03-29T11:56:10.970 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.958+0000 7f984ffff640 10 monclient: _finish_auth 0 2024-03-29T11:56:10.970 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.958+0000 7f984ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:56:10.970 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.958+0000 7f984ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:56:10.970 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.958+0000 7f986de49640 5 monclient: authenticate success, global_id 4497 2024-03-29T11:56:10.970 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.958+0000 7f986de49640 10 monclient: _renew_subs 2024-03-29T11:56:10.970 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.958+0000 7f986de49640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:56:10.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.958+0000 7f986de49640 1 -- 172.21.15.98:0/1577195068 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f986806fc20 con 0x7f98680b0770 2024-03-29T11:56:10.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.958+0000 7f986de49640 10 client.4497.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:56:10.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.958+0000 7f986de49640 10 monclient: _renew_subs 2024-03-29T11:56:10.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.958+0000 7f986de49640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:56:10.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.958+0000 7f986de49640 1 -- 172.21.15.98:0/1577195068 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f986813cfe0 con 0x7f98680b0770 2024-03-29T11:56:10.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.958+0000 7f986de49640 1 librados: init done 2024-03-29T11:56:10.973 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.959+0000 7f984ffff640 1 -- 172.21.15.98:0/1577195068 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f985c004a90 con 0x7f98680b0770 2024-03-29T11:56:10.973 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.959+0000 7f984ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:56:10.973 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.959+0000 7f984ffff640 1 -- 172.21.15.98:0/1577195068 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f985c004e20 con 0x7f98680b0770 2024-03-29T11:56:10.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.959+0000 7f984ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:56:10.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.959+0000 7f984ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:56:10.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.959+0000 7f984ffff640 10 monclient: dump: 2024-03-29T11:56:10.974 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:56:10.974 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:56:10.974 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:10.974 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:10.974 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:56:10.974 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:10.974 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:56:10.974 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:56:10.974 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:56:10.974 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:10.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.959+0000 7f9865ffb640 4 set_mon_vals no callback set 2024-03-29T11:56:10.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.960+0000 7f986de49640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:56:10.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.961+0000 7f986de49640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:56:10.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.961+0000 7f984ffff640 1 -- 172.21.15.98:0/1577195068 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f985c004f80 con 0x7f98680b0770 2024-03-29T11:56:10.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.961+0000 7f986de49640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:56:10.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.961+0000 7f986de49640 1 -- 172.21.15.98:0/1577195068 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9848005080 con 0x7f98680b0770 2024-03-29T11:56:10.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.962+0000 7f984ffff640 1 -- 172.21.15.98:0/1577195068 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f985c084d40 con 0x7f98680b0770 2024-03-29T11:56:10.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.962+0000 7f984ffff640 10 client.4497.objecter ms_dispatch 0x7f9868000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:56:10.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.962+0000 7f984ffff640 3 client.4497.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:56:10.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.962+0000 7f984ffff640 3 client.4497.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:56:10.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.962+0000 7f984ffff640 20 client.4497.objecter dump_active .. 0 homeless 2024-03-29T11:56:10.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.962+0000 7f98677fe640 10 monclient: get_auth_request con 0x7f985406dbf0 auth_method 0 2024-03-29T11:56:10.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.963+0000 7f984ffff640 10 client.4497.objecter ms_handle_connect 0x7f985406dbf0 2024-03-29T11:56:10.978 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.970+0000 7f984ffff640 1 -- 172.21.15.98:0/1577195068 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f985c050620 con 0x7f98680b0770 2024-03-29T11:56:10.978 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.970+0000 7f984ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:56:10.978 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:10.970+0000 7f984ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:56:11.312 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:11.302+0000 7f986de49640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:56:11.313 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:11.302+0000 7f986de49640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:56:11.313 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:11.302+0000 7f986de49640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:56:11.313 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:11.302+0000 7f986de49640 1 -- 172.21.15.98:0/1577195068 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9848005330 con 0x7f98680b0770 2024-03-29T11:56:11.313 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:11.304+0000 7f984ffff640 1 -- 172.21.15.98:0/1577195068 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f985c052550 con 0x7f98680b0770 2024-03-29T11:56:11.313 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:11.304+0000 7f984ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:56:11.313 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:11.304+0000 7f984ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:56:11.313 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:56:11.314 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:56:11.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:11.307+0000 7f986de49640 10 librados: watch_flush enter 2024-03-29T11:56:11.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:11.308+0000 7f986de49640 10 librados: watch_flush exit 2024-03-29T11:56:11.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:11.308+0000 7f986de49640 20 client.4497.objecter shutdown clearing up homeless session... 2024-03-29T11:56:11.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:11.308+0000 7f986de49640 10 client.4497.objecter successfully canceled tick 2024-03-29T11:56:11.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:11.308+0000 7f986de49640 1 -- 172.21.15.98:0/1577195068 >> v1:172.21.15.98:6816/4065186974 conn(0x7f985406dbf0 legacy=0x7f98540700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:11.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:11.308+0000 7f986de49640 10 monclient: shutdown 2024-03-29T11:56:11.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:11.308+0000 7f986de49640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:56:11.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:11.308+0000 7f986de49640 1 -- 172.21.15.98:0/1577195068 >> v1:172.21.15.98:6789/0 conn(0x7f98680b0770 legacy=0x7f98681d69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:11.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:11.308+0000 7f986de49640 1 -- 172.21.15.98:0/1577195068 shutdown_connections 2024-03-29T11:56:11.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:11.308+0000 7f986de49640 1 -- 172.21.15.98:0/1577195068 >> 172.21.15.98:0/1577195068 conn(0x7f986806b660 msgr2=0x7f9868142750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:56:11.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:11.308+0000 7f986de49640 1 -- 172.21.15.98:0/1577195068 shutdown_connections 2024-03-29T11:56:11.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:11.308+0000 7f986de49640 1 -- 172.21.15.98:0/1577195068 wait complete. 2024-03-29T11:56:11.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:11.308+0000 7f986de49640 1 librados: shutdown 2024-03-29T11:56:11.337 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:56:18.143 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:56:18.291 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.282+0000 7f7d17d52640 10 monclient: get_monmap_and_config 2024-03-29T11:56:18.291 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.282+0000 7f7d17d52640 10 monclient: build_initial_monmap 2024-03-29T11:56:18.291 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.282+0000 7f7d17d52640 10 monclient: monmap: 2024-03-29T11:56:18.291 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:56:18.291 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:56:18.291 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:56:18.283181+0000 2024-03-29T11:56:18.291 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:56:18.283181+0000 2024-03-29T11:56:18.291 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:56:18.291 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:18.291 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:56:18.291 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:56:18.291 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:56:18.292 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:18.292 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.283+0000 7f7d17d52640 1 Processor -- start 2024-03-29T11:56:18.292 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.283+0000 7f7d17d52640 1 -- start start 2024-03-29T11:56:18.292 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.283+0000 7f7d17d52640 10 monclient: init 2024-03-29T11:56:18.292 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.284+0000 7f7d17d52640 10 monclient: _reopen_session rank -1 2024-03-29T11:56:18.292 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.284+0000 7f7d17d52640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:56:18.293 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.284+0000 7f7d17d52640 10 monclient(hunting): picked mon.noname-b con 0x7f7d101457f0 addr v1:172.21.15.98:6790/0 2024-03-29T11:56:18.293 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.284+0000 7f7d17d52640 10 monclient(hunting): picked mon.noname-c con 0x7f7d1013f650 addr v1:172.21.15.121:6789/0 2024-03-29T11:56:18.293 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.284+0000 7f7d17d52640 10 monclient(hunting): picked mon.noname-a con 0x7f7d100a9d70 addr v1:172.21.15.98:6789/0 2024-03-29T11:56:18.293 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.284+0000 7f7d17d52640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d100aaef0 con 0x7f7d100a9d70 2024-03-29T11:56:18.293 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.284+0000 7f7d17d52640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d10145bd0 con 0x7f7d1013f650 2024-03-29T11:56:18.293 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.284+0000 7f7d17d52640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d10145eb0 con 0x7f7d101457f0 2024-03-29T11:56:18.293 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.284+0000 7f7d17d52640 10 monclient(hunting): _renew_subs 2024-03-29T11:56:18.293 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.284+0000 7f7d17d52640 10 monclient(hunting): authenticate will time out at 1285.369402s 2024-03-29T11:56:18.294 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.286+0000 7f7d152ed640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f7d1013f650 0x7f7d100b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:38398/0 (socket says 172.21.15.98:38398) 2024-03-29T11:56:18.294 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.286+0000 7f7d152ed640 1 -- 172.21.15.98:0/671866478 learned_addr learned my addr 172.21.15.98:0/671866478 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:56:18.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.288+0000 7f7d14aec640 1 -- 172.21.15.98:0/671866478 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1652881200 0 0) 0x7f7d10145bd0 con 0x7f7d1013f650 2024-03-29T11:56:18.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.289+0000 7f7d14aec640 10 monclient(hunting): _negotiate 2024-03-29T11:56:18.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.289+0000 7f7d14aec640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:56:18.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.289+0000 7f7d14aec640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:56:18.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.289+0000 7f7d14aec640 10 monclient(hunting): my global_id is 4504 2024-03-29T11:56:18.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.289+0000 7f7d14aec640 1 -- 172.21.15.98:0/671866478 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cfc003520 con 0x7f7d1013f650 2024-03-29T11:56:18.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.289+0000 7f7d14aec640 1 -- 172.21.15.98:0/671866478 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2235983215 0 0) 0x7f7cfc003520 con 0x7f7d1013f650 2024-03-29T11:56:18.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.289+0000 7f7d14aec640 1 -- 172.21.15.98:0/671866478 >> v1:172.21.15.98:6790/0 conn(0x7f7d101457f0 legacy=0x7f7d1013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:18.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.289+0000 7f7d14aec640 1 -- 172.21.15.98:0/671866478 >> v1:172.21.15.98:6789/0 conn(0x7f7d100a9d70 legacy=0x7f7d1014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:56:18.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.289+0000 7f7d14aec640 10 monclient: _finish_hunting 0 2024-03-29T11:56:18.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.289+0000 7f7d14aec640 1 monclient: found mon.noname-c 2024-03-29T11:56:18.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.289+0000 7f7d14aec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:56:18.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.289+0000 7f7d14aec640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:56:18.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.289+0000 7f7d14aec640 1 -- 172.21.15.98:0/671866478 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d10146340 con 0x7f7d1013f650 2024-03-29T11:56:18.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.289+0000 7f7d14aec640 10 monclient: _finish_auth 0 2024-03-29T11:56:18.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.289+0000 7f7d14aec640 10 monclient: _check_auth_tickets 2024-03-29T11:56:18.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.289+0000 7f7d14aec640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:56:18.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.289+0000 7f7d14aec640 1 -- 172.21.15.98:0/671866478 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f7d00001690 con 0x7f7d1013f650 2024-03-29T11:56:18.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.289+0000 7f7d14aec640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:56:18.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.289+0000 7f7d14aec640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:56:18.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.289+0000 7f7d14aec640 10 monclient: dump: 2024-03-29T11:56:18.299 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:56:18.299 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:56:18.299 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:18.299 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:18.299 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:56:18.299 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:18.299 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:56:18.299 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:56:18.299 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:56:18.299 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:18.300 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.290+0000 7f7d14aec640 1 -- 172.21.15.98:0/671866478 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d00001920 con 0x7f7d1013f650 2024-03-29T11:56:18.300 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.290+0000 7f7d14aec640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:56:18.300 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.290+0000 7f7d14aec640 1 -- 172.21.15.98:0/671866478 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f7d00001cf0 con 0x7f7d1013f650 2024-03-29T11:56:18.300 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.290+0000 7f7d14aec640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:56:18.301 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.290+0000 7f7d14aec640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:56:18.301 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.290+0000 7f7d14aec640 10 monclient: dump: 2024-03-29T11:56:18.301 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:56:18.301 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:56:18.301 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:18.301 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:18.301 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:56:18.301 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:18.301 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:56:18.301 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:56:18.301 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:56:18.301 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:18.301 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.290+0000 7f7d17d52640 5 monclient: authenticate success, global_id 4504 2024-03-29T11:56:18.301 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.290+0000 7f7d17d52640 10 monclient: get_monmap_and_config success 2024-03-29T11:56:18.301 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.290+0000 7f7d17d52640 4 set_mon_vals no callback set 2024-03-29T11:56:18.301 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.291+0000 7f7d17d52640 10 monclient: shutdown 2024-03-29T11:56:18.301 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.291+0000 7f7d17d52640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:56:18.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.291+0000 7f7d17d52640 1 -- 172.21.15.98:0/671866478 >> v1:172.21.15.121:6789/0 conn(0x7f7d1013f650 legacy=0x7f7d100b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:18.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.291+0000 7f7d17d52640 1 -- 172.21.15.98:0/671866478 shutdown_connections 2024-03-29T11:56:18.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.291+0000 7f7d17d52640 1 -- 172.21.15.98:0/671866478 >> 172.21.15.98:0/671866478 conn(0x7f7d10066a00 msgr2=0x7f7d10066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:56:18.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.292+0000 7f7d17d52640 1 -- 172.21.15.98:0/671866478 shutdown_connections 2024-03-29T11:56:18.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.292+0000 7f7d17d52640 1 -- 172.21.15.98:0/671866478 wait complete. 2024-03-29T11:56:18.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.293+0000 7f7d17d52640 10 monclient: build_initial_monmap 2024-03-29T11:56:18.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.293+0000 7f7d17d52640 10 monclient: monmap: 2024-03-29T11:56:18.303 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:56:18.303 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:56:18.303 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:56:18.303 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:56:18.304 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:56:18.304 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:18.304 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:56:18.304 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:56:18.304 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:56:18.304 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:18.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.293+0000 7f7d17d52640 1 librados: starting msgr at 2024-03-29T11:56:18.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.293+0000 7f7d17d52640 1 librados: starting objecter 2024-03-29T11:56:18.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.293+0000 7f7d17d52640 1 Processor -- start 2024-03-29T11:56:18.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.293+0000 7f7d17d52640 1 -- start start 2024-03-29T11:56:18.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.293+0000 7f7d17d52640 1 librados: setting wanted keys 2024-03-29T11:56:18.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.293+0000 7f7d17d52640 1 librados: calling monclient init 2024-03-29T11:56:18.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.293+0000 7f7d17d52640 10 monclient: init 2024-03-29T11:56:18.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.294+0000 7f7d17d52640 10 monclient: _reopen_session rank -1 2024-03-29T11:56:18.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.294+0000 7f7d17d52640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:56:18.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.294+0000 7f7d17d52640 10 monclient(hunting): picked mon.noname-c con 0x7f7d100a9d70 addr v1:172.21.15.98:6790/0 2024-03-29T11:56:18.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.294+0000 7f7d17d52640 10 monclient(hunting): picked mon.noname-b con 0x7f7d101457f0 addr v1:172.21.15.121:6789/0 2024-03-29T11:56:18.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.294+0000 7f7d17d52640 10 monclient(hunting): picked mon.noname-a con 0x7f7d101df6a0 addr v1:172.21.15.98:6789/0 2024-03-29T11:56:18.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.294+0000 7f7d17d52640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d101dfb30 con 0x7f7d101df6a0 2024-03-29T11:56:18.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.294+0000 7f7d17d52640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d101426a0 con 0x7f7d101457f0 2024-03-29T11:56:18.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.294+0000 7f7d17d52640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d10142830 con 0x7f7d100a9d70 2024-03-29T11:56:18.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.294+0000 7f7d17d52640 10 monclient(hunting): _renew_subs 2024-03-29T11:56:18.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.294+0000 7f7d17d52640 10 monclient(hunting): authenticate will time out at 1285.379351s 2024-03-29T11:56:18.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.295+0000 7f7d152ed640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f7d101457f0 0x7f7d101dba90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:41588/0 (socket says 172.21.15.98:41588) 2024-03-29T11:56:18.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.295+0000 7f7d152ed640 1 -- 172.21.15.98:0/3293089645 learned_addr learned my addr 172.21.15.98:0/3293089645 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:56:18.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.295+0000 7f7d067fc640 10 client.?.objecter ms_handle_connect 0x7f7d101457f0 2024-03-29T11:56:18.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.295+0000 7f7d067fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:56:18.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.295+0000 7f7d067fc640 1 -- 172.21.15.98:0/3293089645 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1163202153 0 0) 0x7f7d101426a0 con 0x7f7d101457f0 2024-03-29T11:56:18.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.295+0000 7f7d067fc640 10 monclient(hunting): _negotiate 2024-03-29T11:56:18.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.295+0000 7f7d067fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:56:18.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.295+0000 7f7d067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:56:18.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.295+0000 7f7d067fc640 10 monclient(hunting): my global_id is 4507 2024-03-29T11:56:18.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.295+0000 7f7d067fc640 1 -- 172.21.15.98:0/3293089645 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cf4004eb0 con 0x7f7d101457f0 2024-03-29T11:56:18.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.296+0000 7f7d067fc640 1 -- 172.21.15.98:0/3293089645 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1654116186 0 0) 0x7f7cf4004eb0 con 0x7f7d101457f0 2024-03-29T11:56:18.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.296+0000 7f7d067fc640 1 -- 172.21.15.98:0/3293089645 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d101426a0 con 0x7f7d101457f0 2024-03-29T11:56:18.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.296+0000 7f7d067fc640 1 -- 172.21.15.98:0/3293089645 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f7d00001650 con 0x7f7d101457f0 2024-03-29T11:56:18.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.296+0000 7f7d067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:56:18.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.296+0000 7f7d067fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:56:18.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.296+0000 7f7d067fc640 10 monclient(hunting): dump: 2024-03-29T11:56:18.307 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:56:18.307 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:56:18.307 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:18.307 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:18.307 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:56:18.307 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:18.307 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:56:18.307 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:56:18.307 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:56:18.307 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:18.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.296+0000 7f7d067fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:56:18.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.296+0000 7f7d067fc640 1 -- 172.21.15.98:0/3293089645 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 65580903 0 0) 0x7f7d101426a0 con 0x7f7d101457f0 2024-03-29T11:56:18.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.297+0000 7f7d067fc640 1 -- 172.21.15.98:0/3293089645 >> v1:172.21.15.98:6790/0 conn(0x7f7d100a9d70 legacy=0x7f7d101def90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:56:18.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.298+0000 7f7d067fc640 1 -- 172.21.15.98:0/3293089645 >> v1:172.21.15.98:6789/0 conn(0x7f7d101df6a0 legacy=0x7f7d101dc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:18.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.298+0000 7f7d067fc640 10 monclient: _finish_hunting 0 2024-03-29T11:56:18.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.298+0000 7f7d067fc640 1 monclient: found mon.b 2024-03-29T11:56:18.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.298+0000 7f7d067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:56:18.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.299+0000 7f7d067fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:56:18.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.299+0000 7f7d067fc640 1 -- 172.21.15.98:0/3293089645 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d101429c0 con 0x7f7d101457f0 2024-03-29T11:56:18.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.299+0000 7f7d067fc640 10 monclient: _finish_auth 0 2024-03-29T11:56:18.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.299+0000 7f7d067fc640 10 monclient: _check_auth_tickets 2024-03-29T11:56:18.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.299+0000 7f7d067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:56:18.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.299+0000 7f7d067fc640 10 client.?.objecter ms_handle_connect 0x7f7d101df6a0 2024-03-29T11:56:18.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.299+0000 7f7d17d52640 5 monclient: authenticate success, global_id 4507 2024-03-29T11:56:18.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.299+0000 7f7d067fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:56:18.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.299+0000 7f7d17d52640 10 monclient: _renew_subs 2024-03-29T11:56:18.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.299+0000 7f7d17d52640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:56:18.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.299+0000 7f7d17d52640 1 -- 172.21.15.98:0/3293089645 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d101e74e0 con 0x7f7d101457f0 2024-03-29T11:56:18.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.299+0000 7f7d17d52640 10 client.4507.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:56:18.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.299+0000 7f7d17d52640 10 monclient: _renew_subs 2024-03-29T11:56:18.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.299+0000 7f7d17d52640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:56:18.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.299+0000 7f7d17d52640 1 -- 172.21.15.98:0/3293089645 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d101e7910 con 0x7f7d101457f0 2024-03-29T11:56:18.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.299+0000 7f7d17d52640 1 librados: init done 2024-03-29T11:56:18.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.300+0000 7f7d17d52640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:56:18.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.300+0000 7f7d17d52640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:56:18.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.300+0000 7f7d17d52640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:56:18.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.300+0000 7f7d17d52640 1 -- 172.21.15.98:0/3293089645 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d10075f10 con 0x7f7d101457f0 2024-03-29T11:56:18.312 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.302+0000 7f7d067fc640 1 -- 172.21.15.98:0/3293089645 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d000011e0 con 0x7f7d101457f0 2024-03-29T11:56:18.312 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.302+0000 7f7d067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:56:18.312 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.302+0000 7f7d07fff640 4 set_mon_vals no callback set 2024-03-29T11:56:18.312 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.303+0000 7f7d067fc640 1 -- 172.21.15.98:0/3293089645 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f7d00004dd0 con 0x7f7d101457f0 2024-03-29T11:56:18.312 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.303+0000 7f7d067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:56:18.312 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.303+0000 7f7d067fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:56:18.312 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.303+0000 7f7d067fc640 10 monclient: dump: 2024-03-29T11:56:18.312 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:56:18.312 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:56:18.312 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:18.312 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:18.312 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:56:18.313 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:18.313 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:56:18.313 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:56:18.313 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:56:18.313 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:18.313 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.304+0000 7f7d067fc640 1 -- 172.21.15.98:0/3293089645 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f7d0001ae20 con 0x7f7d101457f0 2024-03-29T11:56:18.313 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.305+0000 7f7d15aee640 10 monclient: get_auth_request con 0x7f7cf406db00 auth_method 0 2024-03-29T11:56:18.314 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.306+0000 7f7d067fc640 1 -- 172.21.15.98:0/3293089645 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f7d00054a90 con 0x7f7d101457f0 2024-03-29T11:56:18.314 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.306+0000 7f7d067fc640 10 client.4507.objecter ms_dispatch 0x7f7d10000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:56:18.314 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.306+0000 7f7d067fc640 3 client.4507.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:56:18.314 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.306+0000 7f7d067fc640 3 client.4507.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:56:18.314 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.306+0000 7f7d067fc640 20 client.4507.objecter dump_active .. 0 homeless 2024-03-29T11:56:18.314 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.306+0000 7f7d067fc640 10 client.4507.objecter ms_handle_connect 0x7f7cf406db00 2024-03-29T11:56:18.319 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.311+0000 7f7d067fc640 1 -- 172.21.15.98:0/3293089645 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7d00050d30 con 0x7f7d101457f0 2024-03-29T11:56:18.319 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.311+0000 7f7d067fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:56:18.319 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.311+0000 7f7d067fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:56:18.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.631+0000 7f7d17d52640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:56:18.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.631+0000 7f7d17d52640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:56:18.640 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.631+0000 7f7d17d52640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:56:18.640 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.631+0000 7f7d17d52640 1 -- 172.21.15.98:0/3293089645 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d10144910 con 0x7f7d101457f0 2024-03-29T11:56:18.640 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.632+0000 7f7d067fc640 1 -- 172.21.15.98:0/3293089645 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f7d00050d30 con 0x7f7d101457f0 2024-03-29T11:56:18.640 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.632+0000 7f7d067fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:56:18.640 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.632+0000 7f7d067fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:56:18.641 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:56:18.642 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:56:18.645 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.637+0000 7f7d17d52640 10 librados: watch_flush enter 2024-03-29T11:56:18.645 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.637+0000 7f7d17d52640 10 librados: watch_flush exit 2024-03-29T11:56:18.646 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.637+0000 7f7d17d52640 20 client.4507.objecter shutdown clearing up homeless session... 2024-03-29T11:56:18.646 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.637+0000 7f7d17d52640 10 client.4507.objecter successfully canceled tick 2024-03-29T11:56:18.646 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.637+0000 7f7d17d52640 1 -- 172.21.15.98:0/3293089645 >> v1:172.21.15.98:6816/4065186974 conn(0x7f7cf406db00 legacy=0x7f7cf406ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:18.646 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.637+0000 7f7d17d52640 10 monclient: shutdown 2024-03-29T11:56:18.646 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.637+0000 7f7d17d52640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:56:18.646 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.637+0000 7f7d17d52640 1 -- 172.21.15.98:0/3293089645 >> v1:172.21.15.121:6789/0 conn(0x7f7d101457f0 legacy=0x7f7d101dba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:18.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.639+0000 7f7d17d52640 1 -- 172.21.15.98:0/3293089645 shutdown_connections 2024-03-29T11:56:18.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.639+0000 7f7d17d52640 1 -- 172.21.15.98:0/3293089645 >> 172.21.15.98:0/3293089645 conn(0x7f7d10066a00 msgr2=0x7f7d100b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:56:18.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.639+0000 7f7d17d52640 1 -- 172.21.15.98:0/3293089645 shutdown_connections 2024-03-29T11:56:18.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.639+0000 7f7d17d52640 1 -- 172.21.15.98:0/3293089645 wait complete. 2024-03-29T11:56:18.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:18.639+0000 7f7d17d52640 1 librados: shutdown 2024-03-29T11:56:18.664 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:56:25.472 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:56:25.618 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.609+0000 7fa0ed834640 10 monclient: get_monmap_and_config 2024-03-29T11:56:25.618 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.609+0000 7fa0ed834640 10 monclient: build_initial_monmap 2024-03-29T11:56:25.618 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.609+0000 7fa0ed834640 10 monclient: monmap: 2024-03-29T11:56:25.618 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:56:25.618 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:56:25.618 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:56:25.610514+0000 2024-03-29T11:56:25.618 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:56:25.610514+0000 2024-03-29T11:56:25.618 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:56:25.618 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:25.618 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:56:25.618 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:56:25.619 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:56:25.619 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:25.620 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.612+0000 7fa0ed834640 1 Processor -- start 2024-03-29T11:56:25.622 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.614+0000 7fa0ed834640 1 -- start start 2024-03-29T11:56:25.622 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.614+0000 7fa0ed834640 10 monclient: init 2024-03-29T11:56:25.624 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.615+0000 7fa0ed834640 10 monclient: _reopen_session rank -1 2024-03-29T11:56:25.624 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.615+0000 7fa0ed834640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:56:25.624 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.615+0000 7fa0ec832640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fa0e80b2800 0x7fa0e8142620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:46804/0 (socket says 172.21.15.98:46804) 2024-03-29T11:56:25.624 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.615+0000 7fa0ec832640 1 -- 172.21.15.98:0/2896754638 learned_addr learned my addr 172.21.15.98:0/2896754638 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:56:25.624 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.615+0000 7fa0ed834640 10 monclient(hunting): picked mon.noname-b con 0x7fa0e80b2800 addr v1:172.21.15.98:6790/0 2024-03-29T11:56:25.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.615+0000 7fa0ed834640 10 monclient(hunting): picked mon.noname-a con 0x7fa0e80a9c00 addr v1:172.21.15.98:6789/0 2024-03-29T11:56:25.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.615+0000 7fa0ed834640 10 monclient(hunting): picked mon.noname-c con 0x7fa0e80b38c0 addr v1:172.21.15.121:6789/0 2024-03-29T11:56:25.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.615+0000 7fa0ed834640 1 -- 172.21.15.98:0/2896754638 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0e8141eb0 con 0x7fa0e80a9c00 2024-03-29T11:56:25.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.615+0000 7fa0ed834640 1 -- 172.21.15.98:0/2896754638 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0e80b22d0 con 0x7fa0e80b38c0 2024-03-29T11:56:25.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.615+0000 7fa0ed834640 1 -- 172.21.15.98:0/2896754638 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0e8142d30 con 0x7fa0e80b2800 2024-03-29T11:56:25.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.615+0000 7fa0ed834640 10 monclient(hunting): _renew_subs 2024-03-29T11:56:25.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.615+0000 7fa0ed834640 10 monclient(hunting): authenticate will time out at 1292.700868s 2024-03-29T11:56:25.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.616+0000 7fa0e77fe640 1 -- 172.21.15.98:0/2896754638 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3416999202 0 0) 0x7fa0e8142d30 con 0x7fa0e80b2800 2024-03-29T11:56:25.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.616+0000 7fa0e77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:56:25.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.616+0000 7fa0e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:56:25.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.616+0000 7fa0e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:56:25.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.616+0000 7fa0e77fe640 10 monclient(hunting): my global_id is 4532 2024-03-29T11:56:25.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.616+0000 7fa0e77fe640 1 -- 172.21.15.98:0/2896754638 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0d0003520 con 0x7fa0e80b2800 2024-03-29T11:56:25.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.616+0000 7fa0e77fe640 1 -- 172.21.15.98:0/2896754638 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 428556525 0 0) 0x7fa0d0003520 con 0x7fa0e80b2800 2024-03-29T11:56:25.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.616+0000 7fa0e77fe640 1 -- 172.21.15.98:0/2896754638 >> v1:172.21.15.121:6789/0 conn(0x7fa0e80b38c0 legacy=0x7fa0e814d7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:56:25.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.618+0000 7fa0e77fe640 1 -- 172.21.15.98:0/2896754638 >> v1:172.21.15.98:6789/0 conn(0x7fa0e80a9c00 legacy=0x7fa0e813bb60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:56:25.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.619+0000 7fa0e77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:56:25.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.619+0000 7fa0e77fe640 1 monclient: found mon.noname-b 2024-03-29T11:56:25.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.619+0000 7fa0e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:56:25.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.619+0000 7fa0e77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:56:25.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.619+0000 7fa0e77fe640 1 -- 172.21.15.98:0/2896754638 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e80aaf90 con 0x7fa0e80b2800 2024-03-29T11:56:25.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.619+0000 7fa0e77fe640 10 monclient: _finish_auth 0 2024-03-29T11:56:25.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.619+0000 7fa0e77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:56:25.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.619+0000 7fa0e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:56:25.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.619+0000 7fa0ed834640 5 monclient: authenticate success, global_id 4532 2024-03-29T11:56:25.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.619+0000 7fa0e77fe640 1 -- 172.21.15.98:0/2896754638 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fa0d8001690 con 0x7fa0e80b2800 2024-03-29T11:56:25.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.619+0000 7fa0ed834640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:56:25.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.619+0000 7fa0e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:56:25.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.619+0000 7fa0e77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:56:25.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.619+0000 7fa0e77fe640 10 monclient: dump: 2024-03-29T11:56:25.630 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:56:25.630 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:56:25.630 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:25.630 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:25.630 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:56:25.630 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:25.630 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:56:25.630 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:56:25.630 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:56:25.630 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:25.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.619+0000 7fa0ed834640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:56:25.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.622+0000 7fa0e77fe640 1 -- 172.21.15.98:0/2896754638 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0d80017f0 con 0x7fa0e80b2800 2024-03-29T11:56:25.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.622+0000 7fa0e77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:56:25.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.622+0000 7fa0e77fe640 1 -- 172.21.15.98:0/2896754638 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fa0d8001c40 con 0x7fa0e80b2800 2024-03-29T11:56:25.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.622+0000 7fa0e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:56:25.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.622+0000 7fa0e77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:56:25.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.622+0000 7fa0e77fe640 10 monclient: dump: 2024-03-29T11:56:25.632 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:56:25.632 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:56:25.632 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:25.632 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:25.632 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:56:25.632 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:25.632 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:56:25.632 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:56:25.632 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:56:25.632 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:25.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.622+0000 7fa0ed834640 10 monclient: get_monmap_and_config success 2024-03-29T11:56:25.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.622+0000 7fa0ed834640 4 set_mon_vals no callback set 2024-03-29T11:56:25.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.623+0000 7fa0ed834640 10 monclient: shutdown 2024-03-29T11:56:25.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.623+0000 7fa0ed834640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:56:25.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.623+0000 7fa0ed834640 1 -- 172.21.15.98:0/2896754638 >> v1:172.21.15.98:6790/0 conn(0x7fa0e80b2800 legacy=0x7fa0e8142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:25.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.623+0000 7fa0ed834640 1 -- 172.21.15.98:0/2896754638 shutdown_connections 2024-03-29T11:56:25.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.623+0000 7fa0ed834640 1 -- 172.21.15.98:0/2896754638 >> 172.21.15.98:0/2896754638 conn(0x7fa0e8066a00 msgr2=0x7fa0e8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:56:25.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.623+0000 7fa0ed834640 1 -- 172.21.15.98:0/2896754638 shutdown_connections 2024-03-29T11:56:25.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.623+0000 7fa0ed834640 1 -- 172.21.15.98:0/2896754638 wait complete. 2024-03-29T11:56:25.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.624+0000 7fa0ed834640 10 monclient: build_initial_monmap 2024-03-29T11:56:25.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.624+0000 7fa0ed834640 10 monclient: monmap: 2024-03-29T11:56:25.633 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:56:25.634 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:56:25.634 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:56:25.634 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:56:25.634 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:56:25.634 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:25.634 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:56:25.634 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:56:25.634 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:56:25.634 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:25.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.624+0000 7fa0ed834640 1 librados: starting msgr at 2024-03-29T11:56:25.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.624+0000 7fa0ed834640 1 librados: starting objecter 2024-03-29T11:56:25.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.624+0000 7fa0ed834640 1 Processor -- start 2024-03-29T11:56:25.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.627+0000 7fa0ed834640 1 -- start start 2024-03-29T11:56:25.636 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.627+0000 7fa0ed834640 1 librados: setting wanted keys 2024-03-29T11:56:25.636 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.627+0000 7fa0ed834640 1 librados: calling monclient init 2024-03-29T11:56:25.636 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.627+0000 7fa0ed834640 10 monclient: init 2024-03-29T11:56:25.637 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.628+0000 7fa0ed834640 10 monclient: _reopen_session rank -1 2024-03-29T11:56:25.637 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.628+0000 7fa0ed834640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:56:25.640 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.631+0000 7fa0ed834640 10 monclient(hunting): picked mon.noname-a con 0x7fa0e80a9c00 addr v1:172.21.15.98:6789/0 2024-03-29T11:56:25.640 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.631+0000 7fa0ed834640 10 monclient(hunting): picked mon.noname-b con 0x7fa0e80b2800 addr v1:172.21.15.121:6789/0 2024-03-29T11:56:25.640 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.631+0000 7fa0ed834640 10 monclient(hunting): picked mon.noname-c con 0x7fa0e80b38c0 addr v1:172.21.15.98:6790/0 2024-03-29T11:56:25.640 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.631+0000 7fa0ed834640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0e8146100 con 0x7fa0e80a9c00 2024-03-29T11:56:25.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.631+0000 7fa0ed834640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0e81432a0 con 0x7fa0e80b2800 2024-03-29T11:56:25.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.631+0000 7fa0ed834640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0e8143430 con 0x7fa0e80b38c0 2024-03-29T11:56:25.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.631+0000 7fa0ed834640 10 monclient(hunting): _renew_subs 2024-03-29T11:56:25.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.631+0000 7fa0ed834640 10 monclient(hunting): authenticate will time out at 1292.716749s 2024-03-29T11:56:25.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.632+0000 7fa0ed033640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fa0e80b38c0 0x7fa0e81e28c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:50582/0 (socket says 172.21.15.98:50582) 2024-03-29T11:56:25.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.632+0000 7fa0ed033640 1 -- 172.21.15.98:0/117545098 learned_addr learned my addr 172.21.15.98:0/117545098 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:56:25.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.632+0000 7fa0e57fa640 10 client.?.objecter ms_handle_connect 0x7fa0e80b38c0 2024-03-29T11:56:25.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.633+0000 7fa0e57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:56:25.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.633+0000 7fa0e57fa640 10 client.?.objecter ms_handle_connect 0x7fa0e80a9c00 2024-03-29T11:56:25.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.633+0000 7fa0e57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:56:25.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.633+0000 7fa0e57fa640 1 -- 172.21.15.98:0/117545098 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2133823924 0 0) 0x7fa0e8146100 con 0x7fa0e80a9c00 2024-03-29T11:56:25.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.633+0000 7fa0e57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:56:25.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.633+0000 7fa0e57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:56:25.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.633+0000 7fa0e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:56:25.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.633+0000 7fa0e57fa640 10 monclient(hunting): my global_id is 4503 2024-03-29T11:56:25.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.633+0000 7fa0e57fa640 1 -- 172.21.15.98:0/117545098 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0d4004eb0 con 0x7fa0e80a9c00 2024-03-29T11:56:25.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.633+0000 7fa0e57fa640 1 -- 172.21.15.98:0/117545098 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3837476542 0 0) 0x7fa0e0001390 con 0x7fa0e80b38c0 2024-03-29T11:56:25.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.633+0000 7fa0e57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:56:25.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.633+0000 7fa0e57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:56:25.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.633+0000 7fa0e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:56:25.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.633+0000 7fa0e57fa640 10 monclient(hunting): my global_id is 4535 2024-03-29T11:56:25.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.633+0000 7fa0e57fa640 1 -- 172.21.15.98:0/117545098 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0e8146100 con 0x7fa0e80b38c0 2024-03-29T11:56:25.645 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.633+0000 7fa0e57fa640 1 -- 172.21.15.98:0/117545098 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3632966357 0 0) 0x7fa0e8146100 con 0x7fa0e80b38c0 2024-03-29T11:56:25.645 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.633+0000 7fa0e57fa640 1 -- 172.21.15.98:0/117545098 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0e0001390 con 0x7fa0e80b38c0 2024-03-29T11:56:25.645 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.633+0000 7fa0e57fa640 1 -- 172.21.15.98:0/117545098 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fa0e0001a80 con 0x7fa0e80b38c0 2024-03-29T11:56:25.645 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.633+0000 7fa0e57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:56:25.645 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.633+0000 7fa0e57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:56:25.646 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.633+0000 7fa0e57fa640 10 monclient(hunting): dump: 2024-03-29T11:56:25.646 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:56:25.646 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:56:25.646 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:25.646 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:25.646 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:56:25.646 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:25.646 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:56:25.646 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:56:25.646 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:56:25.646 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:25.646 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.633+0000 7fa0e57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:56:25.646 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.634+0000 7fa0e57fa640 1 -- 172.21.15.98:0/117545098 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4070959978 0 0) 0x7fa0e0001390 con 0x7fa0e80b38c0 2024-03-29T11:56:25.646 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.634+0000 7fa0e57fa640 1 -- 172.21.15.98:0/117545098 >> v1:172.21.15.121:6789/0 conn(0x7fa0e80b2800 legacy=0x7fa0e80b0a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:56:25.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.635+0000 7fa0e57fa640 1 -- 172.21.15.98:0/117545098 >> v1:172.21.15.98:6789/0 conn(0x7fa0e80a9c00 legacy=0x7fa0e81daaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:25.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.635+0000 7fa0e57fa640 10 monclient: _finish_hunting 0 2024-03-29T11:56:25.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.635+0000 7fa0e57fa640 1 monclient: found mon.c 2024-03-29T11:56:25.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.635+0000 7fa0e57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:56:25.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.635+0000 7fa0e57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:56:25.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.635+0000 7fa0e57fa640 1 -- 172.21.15.98:0/117545098 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e8145cb0 con 0x7fa0e80b38c0 2024-03-29T11:56:25.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.635+0000 7fa0e57fa640 10 monclient: _finish_auth 0 2024-03-29T11:56:25.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.635+0000 7fa0e57fa640 10 monclient: _check_auth_tickets 2024-03-29T11:56:25.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.635+0000 7fa0e57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:56:25.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.635+0000 7fa0e57fa640 1 -- 172.21.15.98:0/117545098 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0e00020d0 con 0x7fa0e80b38c0 2024-03-29T11:56:25.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.635+0000 7fa0ed834640 5 monclient: authenticate success, global_id 4535 2024-03-29T11:56:25.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.635+0000 7fa0ed834640 10 monclient: _renew_subs 2024-03-29T11:56:25.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.635+0000 7fa0ed834640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:56:25.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.635+0000 7fa0ed834640 1 -- 172.21.15.98:0/117545098 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0e81db1b0 con 0x7fa0e80b38c0 2024-03-29T11:56:25.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.636+0000 7fa0ed834640 10 client.4535.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:56:25.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.636+0000 7fa0e57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:56:25.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.636+0000 7fa0e57fa640 1 -- 172.21.15.98:0/117545098 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fa0e0004d80 con 0x7fa0e80b38c0 2024-03-29T11:56:25.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.636+0000 7fa0e57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:56:25.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.636+0000 7fa0e57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:56:25.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.636+0000 7fa0e57fa640 10 monclient: dump: 2024-03-29T11:56:25.648 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:56:25.648 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:56:25.648 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:25.648 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:25.648 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:56:25.648 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:25.648 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:56:25.648 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:56:25.649 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:56:25.649 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:25.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.636+0000 7fa0ed834640 10 monclient: _renew_subs 2024-03-29T11:56:25.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.636+0000 7fa0ed834640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:56:25.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.636+0000 7fa0ed834640 1 -- 172.21.15.98:0/117545098 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0e80ac1c0 con 0x7fa0e80b38c0 2024-03-29T11:56:25.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.637+0000 7fa0ed834640 1 librados: init done 2024-03-29T11:56:25.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.637+0000 7fa0e6ffd640 4 set_mon_vals no callback set 2024-03-29T11:56:25.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.638+0000 7fa0e57fa640 1 -- 172.21.15.98:0/117545098 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7fa0e001ad50 con 0x7fa0e80b38c0 2024-03-29T11:56:25.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.638+0000 7fa0ed834640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:56:25.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.638+0000 7fa0ed834640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:56:25.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.638+0000 7fa0ed834640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:56:25.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.638+0000 7fa0ed834640 1 -- 172.21.15.98:0/117545098 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0e8145f00 con 0x7fa0e80b38c0 2024-03-29T11:56:25.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.639+0000 7fa0e57fa640 1 -- 172.21.15.98:0/117545098 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7fa0e0085d70 con 0x7fa0e80b38c0 2024-03-29T11:56:25.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.639+0000 7fa0e57fa640 10 client.4535.objecter ms_dispatch 0x7fa0e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:56:25.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.639+0000 7fa0e57fa640 3 client.4535.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:56:25.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.639+0000 7fa0e57fa640 3 client.4535.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:56:25.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.640+0000 7fa0e57fa640 20 client.4535.objecter dump_active .. 0 homeless 2024-03-29T11:56:25.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.640+0000 7fa0ec832640 10 monclient: get_auth_request con 0x7fa0d406dcc0 auth_method 0 2024-03-29T11:56:25.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.640+0000 7fa0e57fa640 10 client.4535.objecter ms_handle_connect 0x7fa0d406dcc0 2024-03-29T11:56:25.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.647+0000 7fa0e57fa640 1 -- 172.21.15.98:0/117545098 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa0e0050620 con 0x7fa0e80b38c0 2024-03-29T11:56:25.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.647+0000 7fa0e57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:56:25.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.647+0000 7fa0e57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:56:25.983 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.975+0000 7fa0ed834640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:56:25.983 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.975+0000 7fa0ed834640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:56:25.983 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.975+0000 7fa0ed834640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:56:25.984 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.975+0000 7fa0ed834640 1 -- 172.21.15.98:0/117545098 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0e81db340 con 0x7fa0e80b38c0 2024-03-29T11:56:25.984 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.976+0000 7fa0e57fa640 1 -- 172.21.15.98:0/117545098 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7fa0e0052550 con 0x7fa0e80b38c0 2024-03-29T11:56:25.984 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.976+0000 7fa0e57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:56:25.984 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.976+0000 7fa0e57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:56:25.984 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:56:25.985 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:56:25.991 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.979+0000 7fa0ed834640 10 librados: watch_flush enter 2024-03-29T11:56:25.991 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.980+0000 7fa0ed834640 10 librados: watch_flush exit 2024-03-29T11:56:25.991 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.980+0000 7fa0ed834640 20 client.4535.objecter shutdown clearing up homeless session... 2024-03-29T11:56:25.991 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.980+0000 7fa0ed834640 10 client.4535.objecter successfully canceled tick 2024-03-29T11:56:25.991 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.980+0000 7fa0ed834640 1 -- 172.21.15.98:0/117545098 >> v1:172.21.15.98:6816/4065186974 conn(0x7fa0d406dcc0 legacy=0x7fa0d4070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:25.991 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.981+0000 7fa0ed834640 10 monclient: shutdown 2024-03-29T11:56:25.992 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.981+0000 7fa0ed834640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:56:25.992 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.981+0000 7fa0ed834640 1 -- 172.21.15.98:0/117545098 >> v1:172.21.15.98:6790/0 conn(0x7fa0e80b38c0 legacy=0x7fa0e81e28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:25.992 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.981+0000 7fa0ed834640 1 -- 172.21.15.98:0/117545098 shutdown_connections 2024-03-29T11:56:25.992 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.981+0000 7fa0ed834640 1 -- 172.21.15.98:0/117545098 >> 172.21.15.98:0/117545098 conn(0x7fa0e8066a00 msgr2=0x7fa0e80b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:56:25.992 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.981+0000 7fa0ed834640 1 -- 172.21.15.98:0/117545098 shutdown_connections 2024-03-29T11:56:25.992 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.981+0000 7fa0ed834640 1 -- 172.21.15.98:0/117545098 wait complete. 2024-03-29T11:56:25.992 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:25.981+0000 7fa0ed834640 1 librados: shutdown 2024-03-29T11:56:26.007 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:56:32.815 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:56:32.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.962+0000 7f8eb629a640 10 monclient: get_monmap_and_config 2024-03-29T11:56:32.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.962+0000 7f8eb629a640 10 monclient: build_initial_monmap 2024-03-29T11:56:32.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.962+0000 7f8eb629a640 10 monclient: monmap: 2024-03-29T11:56:32.971 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:56:32.972 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:56:32.972 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:56:32.963426+0000 2024-03-29T11:56:32.972 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:56:32.963426+0000 2024-03-29T11:56:32.972 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:56:32.972 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:32.972 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:56:32.972 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:56:32.972 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:56:32.972 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:32.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.963+0000 7f8eb629a640 1 Processor -- start 2024-03-29T11:56:32.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.963+0000 7f8eb629a640 1 -- start start 2024-03-29T11:56:32.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.963+0000 7f8eb629a640 10 monclient: init 2024-03-29T11:56:32.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.964+0000 7f8eb629a640 10 monclient: _reopen_session rank -1 2024-03-29T11:56:32.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.964+0000 7f8eb629a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:56:32.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.964+0000 7f8eb629a640 10 monclient(hunting): picked mon.noname-b con 0x7f8eb00b0770 addr v1:172.21.15.98:6790/0 2024-03-29T11:56:32.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.964+0000 7f8eb629a640 10 monclient(hunting): picked mon.noname-c con 0x7f8eb014a700 addr v1:172.21.15.121:6789/0 2024-03-29T11:56:32.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.964+0000 7f8eb629a640 10 monclient(hunting): picked mon.noname-a con 0x7f8eb00ac5a0 addr v1:172.21.15.98:6789/0 2024-03-29T11:56:32.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.964+0000 7f8eb629a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eb00aba70 con 0x7f8eb00ac5a0 2024-03-29T11:56:32.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.964+0000 7f8eb629a640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eb00aecb0 con 0x7f8eb014a700 2024-03-29T11:56:32.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.964+0000 7f8eb629a640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eb00af150 con 0x7f8eb00b0770 2024-03-29T11:56:32.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.964+0000 7f8eb629a640 10 monclient(hunting): _renew_subs 2024-03-29T11:56:32.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.964+0000 7f8eb629a640 10 monclient(hunting): authenticate will time out at 1300.049736s 2024-03-29T11:56:32.978 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.969+0000 7f8eb4837640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f8eb00ac5a0 0x7f8eb0151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57663/0 (socket says 172.21.15.98:57663) 2024-03-29T11:56:32.978 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.969+0000 7f8eb4837640 1 -- 172.21.15.98:0/3020425997 learned_addr learned my addr 172.21.15.98:0/3020425997 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:56:32.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.970+0000 7f8eaeffd640 1 -- 172.21.15.98:0/3020425997 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3489873747 0 0) 0x7f8eb00af150 con 0x7f8eb00b0770 2024-03-29T11:56:32.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.970+0000 7f8eaeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:56:32.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.970+0000 7f8eaeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:56:32.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.970+0000 7f8eaeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:56:32.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.970+0000 7f8eaeffd640 10 monclient(hunting): my global_id is 4538 2024-03-29T11:56:32.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.970+0000 7f8eaeffd640 1 -- 172.21.15.98:0/3020425997 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e98003520 con 0x7f8eb00b0770 2024-03-29T11:56:32.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.970+0000 7f8eaeffd640 1 -- 172.21.15.98:0/3020425997 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2741280030 0 0) 0x7f8e98003520 con 0x7f8eb00b0770 2024-03-29T11:56:32.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.970+0000 7f8eaeffd640 1 -- 172.21.15.98:0/3020425997 >> v1:172.21.15.121:6789/0 conn(0x7f8eb014a700 legacy=0x7f8eb00abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:56:32.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.971+0000 7f8eaeffd640 1 -- 172.21.15.98:0/3020425997 >> v1:172.21.15.98:6789/0 conn(0x7f8eb00ac5a0 legacy=0x7f8eb0151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:32.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.971+0000 7f8eaeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:56:32.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.971+0000 7f8eaeffd640 1 monclient: found mon.noname-b 2024-03-29T11:56:32.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.971+0000 7f8eaeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:56:32.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.971+0000 7f8eaeffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:56:32.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.971+0000 7f8eaeffd640 1 -- 172.21.15.98:0/3020425997 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb00ac9d0 con 0x7f8eb00b0770 2024-03-29T11:56:32.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.971+0000 7f8eaeffd640 10 monclient: _finish_auth 0 2024-03-29T11:56:32.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.971+0000 7f8eaeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:56:32.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.971+0000 7f8eaeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:56:32.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.971+0000 7f8eb629a640 5 monclient: authenticate success, global_id 4538 2024-03-29T11:56:32.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.971+0000 7f8eb629a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:56:32.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.971+0000 7f8eaeffd640 1 -- 172.21.15.98:0/3020425997 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f8ea8001940 con 0x7f8eb00b0770 2024-03-29T11:56:32.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.971+0000 7f8eaeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:56:32.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.971+0000 7f8eaeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:56:32.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.971+0000 7f8eaeffd640 10 monclient: dump: 2024-03-29T11:56:32.986 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:56:32.986 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:56:32.986 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:32.986 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:32.986 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:56:32.986 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:32.986 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:56:32.987 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:56:32.987 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:56:32.987 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:32.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.971+0000 7f8eb629a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:56:32.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.971+0000 7f8eaeffd640 1 -- 172.21.15.98:0/3020425997 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ea8001bd0 con 0x7f8eb00b0770 2024-03-29T11:56:32.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.971+0000 7f8eaeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:56:32.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.971+0000 7f8eb629a640 10 monclient: get_monmap_and_config success 2024-03-29T11:56:32.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.971+0000 7f8eb629a640 4 set_mon_vals no callback set 2024-03-29T11:56:32.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.972+0000 7f8eb629a640 10 monclient: shutdown 2024-03-29T11:56:32.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.972+0000 7f8eb629a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:56:32.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.972+0000 7f8eb629a640 1 -- 172.21.15.98:0/3020425997 >> v1:172.21.15.98:6790/0 conn(0x7f8eb00b0770 legacy=0x7f8eb0149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:32.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.975+0000 7f8eb629a640 1 -- 172.21.15.98:0/3020425997 shutdown_connections 2024-03-29T11:56:32.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.975+0000 7f8eb629a640 1 -- 172.21.15.98:0/3020425997 >> 172.21.15.98:0/3020425997 conn(0x7f8eb006b660 msgr2=0x7f8eb006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:56:32.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.975+0000 7f8eb629a640 1 -- 172.21.15.98:0/3020425997 shutdown_connections 2024-03-29T11:56:32.988 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.976+0000 7f8eb629a640 1 -- 172.21.15.98:0/3020425997 wait complete. 2024-03-29T11:56:32.988 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.977+0000 7f8eb629a640 10 monclient: build_initial_monmap 2024-03-29T11:56:32.988 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.977+0000 7f8eb629a640 10 monclient: monmap: 2024-03-29T11:56:32.989 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:56:32.989 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:56:32.989 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:56:32.989 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:56:32.989 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:56:32.989 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:32.989 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:56:32.989 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:56:32.989 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:56:32.989 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:32.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.977+0000 7f8eb629a640 1 librados: starting msgr at 2024-03-29T11:56:32.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.977+0000 7f8eb629a640 1 librados: starting objecter 2024-03-29T11:56:32.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.977+0000 7f8eb629a640 1 Processor -- start 2024-03-29T11:56:32.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.978+0000 7f8eb629a640 1 -- start start 2024-03-29T11:56:32.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.978+0000 7f8eb629a640 1 librados: setting wanted keys 2024-03-29T11:56:32.990 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.978+0000 7f8eb629a640 1 librados: calling monclient init 2024-03-29T11:56:32.990 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.978+0000 7f8eb629a640 10 monclient: init 2024-03-29T11:56:32.990 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.978+0000 7f8eb629a640 10 monclient: _reopen_session rank -1 2024-03-29T11:56:32.990 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.978+0000 7f8eb629a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:56:32.990 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.979+0000 7f8eb629a640 10 monclient(hunting): picked mon.noname-a con 0x7f8eb00ac5a0 addr v1:172.21.15.98:6789/0 2024-03-29T11:56:32.990 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.979+0000 7f8eb629a640 10 monclient(hunting): picked mon.noname-b con 0x7f8eb00b0770 addr v1:172.21.15.121:6789/0 2024-03-29T11:56:32.990 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.979+0000 7f8eb629a640 10 monclient(hunting): picked mon.noname-c con 0x7f8eb014a700 addr v1:172.21.15.98:6790/0 2024-03-29T11:56:32.990 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.979+0000 7f8eb629a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eb01de740 con 0x7f8eb00ac5a0 2024-03-29T11:56:32.990 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.979+0000 7f8eb629a640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eb006f180 con 0x7f8eb00b0770 2024-03-29T11:56:32.990 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.979+0000 7f8eb629a640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eb006f3f0 con 0x7f8eb014a700 2024-03-29T11:56:32.990 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.979+0000 7f8eb629a640 10 monclient(hunting): _renew_subs 2024-03-29T11:56:32.990 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.979+0000 7f8eb629a640 10 monclient(hunting): authenticate will time out at 1300.064170s 2024-03-29T11:56:32.991 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.982+0000 7f8eaffff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f8eb00ac5a0 0x7f8eb0144bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57859/0 (socket says 172.21.15.98:57859) 2024-03-29T11:56:32.991 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.982+0000 7f8eaffff640 1 -- 172.21.15.98:0/3402895907 learned_addr learned my addr 172.21.15.98:0/3402895907 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:56:32.991 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.983+0000 7f8eacff9640 10 client.?.objecter ms_handle_connect 0x7f8eb014a700 2024-03-29T11:56:32.991 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.983+0000 7f8eacff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:56:32.993 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.985+0000 7f8eacff9640 10 client.?.objecter ms_handle_connect 0x7f8eb00ac5a0 2024-03-29T11:56:32.993 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.985+0000 7f8eacff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:56:32.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.987+0000 7f8eacff9640 10 client.?.objecter ms_handle_connect 0x7f8eb00b0770 2024-03-29T11:56:32.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.987+0000 7f8eacff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:56:32.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.987+0000 7f8eacff9640 1 -- 172.21.15.98:0/3402895907 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2748519801 0 0) 0x7f8eb01de740 con 0x7f8eb00ac5a0 2024-03-29T11:56:32.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.987+0000 7f8eacff9640 10 monclient(hunting): _negotiate 2024-03-29T11:56:32.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.987+0000 7f8eacff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:56:32.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.987+0000 7f8eacff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:56:32.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.987+0000 7f8eacff9640 10 monclient(hunting): my global_id is 4509 2024-03-29T11:56:32.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.987+0000 7f8eacff9640 1 -- 172.21.15.98:0/3402895907 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e9c004eb0 con 0x7f8eb00ac5a0 2024-03-29T11:56:32.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.987+0000 7f8eacff9640 1 -- 172.21.15.98:0/3402895907 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1526020668 0 0) 0x7f8e9c004eb0 con 0x7f8eb00ac5a0 2024-03-29T11:56:32.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.987+0000 7f8eacff9640 1 -- 172.21.15.98:0/3402895907 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8eb01de740 con 0x7f8eb00ac5a0 2024-03-29T11:56:32.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.987+0000 7f8eacff9640 1 -- 172.21.15.98:0/3402895907 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f8ea8003b50 con 0x7f8eb00ac5a0 2024-03-29T11:56:32.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.987+0000 7f8eacff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:56:32.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.987+0000 7f8eacff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:56:32.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.987+0000 7f8eacff9640 10 monclient(hunting): dump: 2024-03-29T11:56:32.997 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:56:32.998 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:56:32.998 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:32.998 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:32.998 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:56:32.998 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:32.998 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:56:32.998 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:56:32.998 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:56:32.998 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:32.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.987+0000 7f8eacff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:56:33.000 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.988+0000 7f8eacff9640 1 -- 172.21.15.98:0/3402895907 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2727232483 0 0) 0x7f8eb01de740 con 0x7f8eb00ac5a0 2024-03-29T11:56:33.000 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.988+0000 7f8eacff9640 1 -- 172.21.15.98:0/3402895907 >> v1:172.21.15.98:6790/0 conn(0x7f8eb014a700 legacy=0x7f8eb013dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:33.000 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.988+0000 7f8eacff9640 1 -- 172.21.15.98:0/3402895907 >> v1:172.21.15.121:6789/0 conn(0x7f8eb00b0770 legacy=0x7f8eb01debd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:33.000 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.988+0000 7f8eacff9640 10 monclient: _finish_hunting 0 2024-03-29T11:56:33.000 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.988+0000 7f8eacff9640 1 monclient: found mon.a 2024-03-29T11:56:33.000 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.988+0000 7f8eacff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:56:33.000 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.988+0000 7f8eacff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:56:33.000 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.988+0000 7f8eacff9640 1 -- 172.21.15.98:0/3402895907 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb0142170 con 0x7f8eb00ac5a0 2024-03-29T11:56:33.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.988+0000 7f8eacff9640 10 monclient: _finish_auth 0 2024-03-29T11:56:33.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.988+0000 7f8eacff9640 10 monclient: _check_auth_tickets 2024-03-29T11:56:33.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.988+0000 7f8eacff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:56:33.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.988+0000 7f8eacff9640 1 -- 172.21.15.98:0/3402895907 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ea8004060 con 0x7f8eb00ac5a0 2024-03-29T11:56:33.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.988+0000 7f8eacff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:56:33.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.988+0000 7f8eacff9640 1 -- 172.21.15.98:0/3402895907 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f8ea80044b0 con 0x7f8eb00ac5a0 2024-03-29T11:56:33.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.988+0000 7f8eacff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:56:33.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.988+0000 7f8eacff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:56:33.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.988+0000 7f8eacff9640 10 monclient: dump: 2024-03-29T11:56:33.001 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:56:33.001 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:56:33.001 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:33.001 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:33.001 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:56:33.002 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:33.002 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:56:33.002 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:56:33.002 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:56:33.002 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:33.002 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.990+0000 7f8eb629a640 5 monclient: authenticate success, global_id 4509 2024-03-29T11:56:33.002 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.990+0000 7f8eb629a640 10 monclient: _renew_subs 2024-03-29T11:56:33.002 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.990+0000 7f8eb629a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:56:33.002 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.990+0000 7f8eb629a640 1 -- 172.21.15.98:0/3402895907 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eb01df2e0 con 0x7f8eb00ac5a0 2024-03-29T11:56:33.002 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.990+0000 7f8eb629a640 10 client.4509.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:56:33.002 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.990+0000 7f8eb629a640 10 monclient: _renew_subs 2024-03-29T11:56:33.002 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.990+0000 7f8eb629a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:56:33.002 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.990+0000 7f8eb629a640 1 -- 172.21.15.98:0/3402895907 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eb013e6d0 con 0x7f8eb00ac5a0 2024-03-29T11:56:33.002 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.990+0000 7f8eb629a640 1 librados: init done 2024-03-29T11:56:33.003 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.990+0000 7f8eadffb640 4 set_mon_vals no callback set 2024-03-29T11:56:33.003 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.991+0000 7f8eb629a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:56:33.003 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.991+0000 7f8eb629a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:56:33.003 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.992+0000 7f8eb629a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:56:33.003 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.992+0000 7f8eb629a640 1 -- 172.21.15.98:0/3402895907 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e90005080 con 0x7f8eb00ac5a0 2024-03-29T11:56:33.004 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.993+0000 7f8eacff9640 1 -- 172.21.15.98:0/3402895907 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f8ea8001dc0 con 0x7f8eb00ac5a0 2024-03-29T11:56:33.004 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.994+0000 7f8eacff9640 1 -- 172.21.15.98:0/3402895907 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f8ea8085020 con 0x7f8eb00ac5a0 2024-03-29T11:56:33.004 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.994+0000 7f8eacff9640 10 client.4509.objecter ms_dispatch 0x7f8eb0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:56:33.004 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.994+0000 7f8eacff9640 3 client.4509.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:56:33.004 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.994+0000 7f8eacff9640 3 client.4509.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:56:33.004 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.994+0000 7f8eacff9640 20 client.4509.objecter dump_active .. 0 homeless 2024-03-29T11:56:33.004 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.994+0000 7f8eaf7fe640 10 monclient: get_auth_request con 0x7f8e9c06dc90 auth_method 0 2024-03-29T11:56:33.004 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:32.994+0000 7f8eacff9640 10 client.4509.objecter ms_handle_connect 0x7f8e9c06dc90 2024-03-29T11:56:33.008 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:33.000+0000 7f8eacff9640 1 -- 172.21.15.98:0/3402895907 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8ea8050900 con 0x7f8eb00ac5a0 2024-03-29T11:56:33.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:33.000+0000 7f8eacff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:56:33.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:33.001+0000 7f8eacff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:56:33.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:33.323+0000 7f8eb629a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:56:33.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:33.323+0000 7f8eb629a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:56:33.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:33.323+0000 7f8eb629a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:56:33.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:33.323+0000 7f8eb629a640 1 -- 172.21.15.98:0/3402895907 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e90005330 con 0x7f8eb00ac5a0 2024-03-29T11:56:33.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:33.324+0000 7f8eacff9640 1 -- 172.21.15.98:0/3402895907 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f8ea8052830 con 0x7f8eb00ac5a0 2024-03-29T11:56:33.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:33.324+0000 7f8eacff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:56:33.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:33.324+0000 7f8eacff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:56:33.334 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:56:33.334 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:56:33.337 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:33.328+0000 7f8eb629a640 10 librados: watch_flush enter 2024-03-29T11:56:33.337 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:33.328+0000 7f8eb629a640 10 librados: watch_flush exit 2024-03-29T11:56:33.337 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:33.328+0000 7f8eb629a640 20 client.4509.objecter shutdown clearing up homeless session... 2024-03-29T11:56:33.337 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:33.328+0000 7f8eb629a640 10 client.4509.objecter successfully canceled tick 2024-03-29T11:56:33.338 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:33.329+0000 7f8eb629a640 1 -- 172.21.15.98:0/3402895907 >> v1:172.21.15.98:6816/4065186974 conn(0x7f8e9c06dc90 legacy=0x7f8e9c070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:33.338 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:33.329+0000 7f8eb629a640 10 monclient: shutdown 2024-03-29T11:56:33.338 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:33.329+0000 7f8eb629a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:56:33.338 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:33.329+0000 7f8eb629a640 1 -- 172.21.15.98:0/3402895907 >> v1:172.21.15.98:6789/0 conn(0x7f8eb00ac5a0 legacy=0x7f8eb0144bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:33.338 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:33.329+0000 7f8eb629a640 1 -- 172.21.15.98:0/3402895907 shutdown_connections 2024-03-29T11:56:33.338 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:33.329+0000 7f8eb629a640 1 -- 172.21.15.98:0/3402895907 >> 172.21.15.98:0/3402895907 conn(0x7f8eb006b660 msgr2=0x7f8eb0149130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:56:33.338 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:33.329+0000 7f8eb629a640 1 -- 172.21.15.98:0/3402895907 shutdown_connections 2024-03-29T11:56:33.338 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:33.329+0000 7f8eb629a640 1 -- 172.21.15.98:0/3402895907 wait complete. 2024-03-29T11:56:33.338 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:33.329+0000 7f8eb629a640 1 librados: shutdown 2024-03-29T11:56:33.355 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:56:40.162 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:56:40.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.301+0000 7f2a9f406640 10 monclient: get_monmap_and_config 2024-03-29T11:56:40.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.301+0000 7f2a9f406640 10 monclient: build_initial_monmap 2024-03-29T11:56:40.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.301+0000 7f2a9f406640 10 monclient: monmap: 2024-03-29T11:56:40.311 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:56:40.311 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:56:40.311 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:56:40.302320+0000 2024-03-29T11:56:40.311 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:56:40.302320+0000 2024-03-29T11:56:40.311 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:56:40.311 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:40.311 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:56:40.311 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:56:40.311 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:56:40.311 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:40.319 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.302+0000 7f2a9f406640 1 Processor -- start 2024-03-29T11:56:40.319 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.302+0000 7f2a9f406640 1 -- start start 2024-03-29T11:56:40.319 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.302+0000 7f2a9f406640 10 monclient: init 2024-03-29T11:56:40.319 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.303+0000 7f2a9f406640 10 monclient: _reopen_session rank -1 2024-03-29T11:56:40.319 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.303+0000 7f2a9f406640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:56:40.320 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.303+0000 7f2a9f406640 10 monclient(hunting): picked mon.noname-a con 0x7f2a980a9930 addr v1:172.21.15.98:6789/0 2024-03-29T11:56:40.320 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.303+0000 7f2a9f406640 10 monclient(hunting): picked mon.noname-c con 0x7f2a98149dc0 addr v1:172.21.15.121:6789/0 2024-03-29T11:56:40.320 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.303+0000 7f2a9f406640 10 monclient(hunting): picked mon.noname-b con 0x7f2a9814ff40 addr v1:172.21.15.98:6790/0 2024-03-29T11:56:40.320 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.303+0000 7f2a9f406640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a980aae60 con 0x7f2a980a9930 2024-03-29T11:56:40.320 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.303+0000 7f2a9f406640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a980b4310 con 0x7f2a98149dc0 2024-03-29T11:56:40.320 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.303+0000 7f2a9f406640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a980afdd0 con 0x7f2a9814ff40 2024-03-29T11:56:40.320 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.303+0000 7f2a9f406640 10 monclient(hunting): _renew_subs 2024-03-29T11:56:40.320 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.303+0000 7f2a9f406640 10 monclient(hunting): authenticate will time out at 1307.388873s 2024-03-29T11:56:40.320 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.306+0000 7f2a9d1a2640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f2a980a9930 0x7f2a98143a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:59029/0 (socket says 172.21.15.98:59029) 2024-03-29T11:56:40.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.306+0000 7f2a9d1a2640 1 -- 172.21.15.98:0/3410852610 learned_addr learned my addr 172.21.15.98:0/3410852610 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:56:40.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.311+0000 7f2a8ffff640 1 -- 172.21.15.98:0/3410852610 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4050074955 0 0) 0x7f2a980b4310 con 0x7f2a98149dc0 2024-03-29T11:56:40.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.311+0000 7f2a8ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:56:40.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.311+0000 7f2a8ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:56:40.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.311+0000 7f2a8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:56:40.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.311+0000 7f2a8ffff640 10 monclient(hunting): my global_id is 4513 2024-03-29T11:56:40.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.311+0000 7f2a8ffff640 1 -- 172.21.15.98:0/3410852610 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a84003520 con 0x7f2a98149dc0 2024-03-29T11:56:40.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.311+0000 7f2a8ffff640 1 -- 172.21.15.98:0/3410852610 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2884205550 0 0) 0x7f2a980afdd0 con 0x7f2a9814ff40 2024-03-29T11:56:40.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.311+0000 7f2a8ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:56:40.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.311+0000 7f2a8ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:56:40.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.311+0000 7f2a8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:56:40.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.311+0000 7f2a8ffff640 10 monclient(hunting): my global_id is 4544 2024-03-29T11:56:40.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.311+0000 7f2a8ffff640 1 -- 172.21.15.98:0/3410852610 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a980b4310 con 0x7f2a9814ff40 2024-03-29T11:56:40.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.311+0000 7f2a8ffff640 1 -- 172.21.15.98:0/3410852610 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2305028446 0 0) 0x7f2a980aae60 con 0x7f2a980a9930 2024-03-29T11:56:40.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.311+0000 7f2a8ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:56:40.322 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.311+0000 7f2a8ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:56:40.322 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.311+0000 7f2a8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:56:40.322 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.311+0000 7f2a8ffff640 10 monclient(hunting): my global_id is 4512 2024-03-29T11:56:40.322 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.311+0000 7f2a8ffff640 1 -- 172.21.15.98:0/3410852610 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a980afdd0 con 0x7f2a980a9930 2024-03-29T11:56:40.322 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.313+0000 7f2a8ffff640 1 -- 172.21.15.98:0/3410852610 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4123570675 0 0) 0x7f2a980b4310 con 0x7f2a9814ff40 2024-03-29T11:56:40.322 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.313+0000 7f2a8ffff640 1 -- 172.21.15.98:0/3410852610 >> v1:172.21.15.121:6789/0 conn(0x7f2a98149dc0 legacy=0x7f2a9814a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:40.322 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.313+0000 7f2a8ffff640 1 -- 172.21.15.98:0/3410852610 >> v1:172.21.15.98:6789/0 conn(0x7f2a980a9930 legacy=0x7f2a98143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:40.322 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.313+0000 7f2a8ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:56:40.322 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.313+0000 7f2a8ffff640 1 monclient: found mon.noname-b 2024-03-29T11:56:40.323 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.313+0000 7f2a8ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:56:40.323 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.313+0000 7f2a8ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:56:40.323 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.313+0000 7f2a8ffff640 1 -- 172.21.15.98:0/3410852610 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a9814a950 con 0x7f2a9814ff40 2024-03-29T11:56:40.323 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.313+0000 7f2a8ffff640 10 monclient: _finish_auth 0 2024-03-29T11:56:40.323 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.313+0000 7f2a8ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:56:40.323 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.313+0000 7f2a8ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:56:40.323 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.314+0000 7f2a9f406640 5 monclient: authenticate success, global_id 4544 2024-03-29T11:56:40.324 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.314+0000 7f2a9f406640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:56:40.324 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.314+0000 7f2a8ffff640 1 -- 172.21.15.98:0/3410852610 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f2a94001520 con 0x7f2a9814ff40 2024-03-29T11:56:40.324 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.314+0000 7f2a8ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:56:40.324 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.314+0000 7f2a8ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:56:40.324 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.314+0000 7f2a8ffff640 10 monclient: dump: 2024-03-29T11:56:40.324 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:56:40.324 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:56:40.324 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:40.324 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:40.324 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:56:40.324 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:40.324 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:56:40.324 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:56:40.324 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:56:40.325 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:40.325 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.314+0000 7f2a9f406640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:56:40.326 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.315+0000 7f2a8ffff640 1 -- 172.21.15.98:0/3410852610 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a94001740 con 0x7f2a9814ff40 2024-03-29T11:56:40.326 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.315+0000 7f2a8ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:56:40.326 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.315+0000 7f2a9f406640 10 monclient: get_monmap_and_config success 2024-03-29T11:56:40.326 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.315+0000 7f2a9f406640 4 set_mon_vals no callback set 2024-03-29T11:56:40.326 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.315+0000 7f2a8ffff640 1 -- 172.21.15.98:0/3410852610 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f2a94001ad0 con 0x7f2a9814ff40 2024-03-29T11:56:40.327 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.315+0000 7f2a8ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:56:40.327 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.315+0000 7f2a8ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:56:40.327 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.315+0000 7f2a8ffff640 10 monclient: dump: 2024-03-29T11:56:40.327 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:56:40.327 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:56:40.327 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:40.327 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:40.327 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:56:40.327 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:40.327 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:56:40.327 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:56:40.327 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:56:40.327 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:40.328 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.316+0000 7f2a9f406640 10 monclient: shutdown 2024-03-29T11:56:40.328 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.316+0000 7f2a9f406640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:56:40.328 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.316+0000 7f2a9f406640 1 -- 172.21.15.98:0/3410852610 >> v1:172.21.15.98:6790/0 conn(0x7f2a9814ff40 legacy=0x7f2a98152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:40.328 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.316+0000 7f2a9f406640 1 -- 172.21.15.98:0/3410852610 shutdown_connections 2024-03-29T11:56:40.328 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.316+0000 7f2a9f406640 1 -- 172.21.15.98:0/3410852610 >> 172.21.15.98:0/3410852610 conn(0x7f2a980b35e0 msgr2=0x7f2a980b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:56:40.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.317+0000 7f2a9f406640 1 -- 172.21.15.98:0/3410852610 shutdown_connections 2024-03-29T11:56:40.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.317+0000 7f2a9f406640 1 -- 172.21.15.98:0/3410852610 wait complete. 2024-03-29T11:56:40.331 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.318+0000 7f2a9f406640 10 monclient: build_initial_monmap 2024-03-29T11:56:40.331 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.318+0000 7f2a9f406640 10 monclient: monmap: 2024-03-29T11:56:40.331 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:56:40.331 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:56:40.331 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:56:40.331 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:56:40.331 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:56:40.331 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:40.331 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:56:40.331 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:56:40.331 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:56:40.331 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:40.331 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.319+0000 7f2a9f406640 1 librados: starting msgr at 2024-03-29T11:56:40.331 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.319+0000 7f2a9f406640 1 librados: starting objecter 2024-03-29T11:56:40.331 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.319+0000 7f2a9f406640 1 Processor -- start 2024-03-29T11:56:40.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.319+0000 7f2a9f406640 1 -- start start 2024-03-29T11:56:40.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.319+0000 7f2a9f406640 1 librados: setting wanted keys 2024-03-29T11:56:40.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.319+0000 7f2a9f406640 1 librados: calling monclient init 2024-03-29T11:56:40.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.319+0000 7f2a9f406640 10 monclient: init 2024-03-29T11:56:40.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.320+0000 7f2a9f406640 10 monclient: _reopen_session rank -1 2024-03-29T11:56:40.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.320+0000 7f2a9f406640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:56:40.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.320+0000 7f2a9f406640 10 monclient(hunting): picked mon.noname-b con 0x7f2a980a9930 addr v1:172.21.15.121:6789/0 2024-03-29T11:56:40.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.320+0000 7f2a9f406640 10 monclient(hunting): picked mon.noname-c con 0x7f2a98149dc0 addr v1:172.21.15.98:6790/0 2024-03-29T11:56:40.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.320+0000 7f2a9f406640 10 monclient(hunting): picked mon.noname-a con 0x7f2a9814ff40 addr v1:172.21.15.98:6789/0 2024-03-29T11:56:40.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.320+0000 7f2a9f406640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a9814aae0 con 0x7f2a9814ff40 2024-03-29T11:56:40.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.320+0000 7f2a9f406640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a98145480 con 0x7f2a980a9930 2024-03-29T11:56:40.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.320+0000 7f2a9f406640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a98144990 con 0x7f2a98149dc0 2024-03-29T11:56:40.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.320+0000 7f2a9f406640 10 monclient(hunting): _renew_subs 2024-03-29T11:56:40.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.320+0000 7f2a9f406640 10 monclient(hunting): authenticate will time out at 1307.405310s 2024-03-29T11:56:40.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.324+0000 7f2a9d1a2640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f2a980a9930 0x7f2a98130280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:43583/0 (socket says 172.21.15.98:43583) 2024-03-29T11:56:40.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.324+0000 7f2a9d1a2640 1 -- 172.21.15.98:0/2983600879 learned_addr learned my addr 172.21.15.98:0/2983600879 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:56:40.339 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.326+0000 7f2a8dffb640 10 client.?.objecter ms_handle_connect 0x7f2a980a9930 2024-03-29T11:56:40.339 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.326+0000 7f2a8dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:56:40.339 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.326+0000 7f2a8dffb640 1 -- 172.21.15.98:0/2983600879 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1588706272 0 0) 0x7f2a98145480 con 0x7f2a980a9930 2024-03-29T11:56:40.339 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.326+0000 7f2a8dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:56:40.339 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.326+0000 7f2a8dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:56:40.339 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.326+0000 7f2a8dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:56:40.340 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.326+0000 7f2a8dffb640 10 monclient(hunting): my global_id is 4516 2024-03-29T11:56:40.340 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.326+0000 7f2a8dffb640 1 -- 172.21.15.98:0/2983600879 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a7c004eb0 con 0x7f2a980a9930 2024-03-29T11:56:40.340 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.326+0000 7f2a8dffb640 10 client.?.objecter ms_handle_connect 0x7f2a98149dc0 2024-03-29T11:56:40.340 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.326+0000 7f2a8dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:56:40.340 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.326+0000 7f2a8dffb640 10 client.?.objecter ms_handle_connect 0x7f2a9814ff40 2024-03-29T11:56:40.340 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.326+0000 7f2a8dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:56:40.340 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.329+0000 7f2a8dffb640 1 -- 172.21.15.98:0/2983600879 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2197749408 0 0) 0x7f2a7c004eb0 con 0x7f2a980a9930 2024-03-29T11:56:40.340 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.330+0000 7f2a8dffb640 1 -- 172.21.15.98:0/2983600879 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a98145480 con 0x7f2a980a9930 2024-03-29T11:56:40.340 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.330+0000 7f2a8dffb640 1 -- 172.21.15.98:0/2983600879 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f2a88001870 con 0x7f2a980a9930 2024-03-29T11:56:40.340 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.330+0000 7f2a8dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:56:40.340 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.330+0000 7f2a8dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:56:40.340 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.330+0000 7f2a8dffb640 10 monclient(hunting): dump: 2024-03-29T11:56:40.340 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:56:40.340 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:56:40.341 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:40.341 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:40.341 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:56:40.341 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:40.341 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:56:40.341 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:56:40.341 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:56:40.341 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:40.341 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.330+0000 7f2a8dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:56:40.341 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.330+0000 7f2a8dffb640 1 -- 172.21.15.98:0/2983600879 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4051581450 0 0) 0x7f2a98144990 con 0x7f2a98149dc0 2024-03-29T11:56:40.341 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.330+0000 7f2a8dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:56:40.341 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.330+0000 7f2a8dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:56:40.341 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.330+0000 7f2a8dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:56:40.341 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.330+0000 7f2a8dffb640 10 monclient(hunting): my global_id is 4547 2024-03-29T11:56:40.341 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.330+0000 7f2a8dffb640 1 -- 172.21.15.98:0/2983600879 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a7c004eb0 con 0x7f2a98149dc0 2024-03-29T11:56:40.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.330+0000 7f2a8dffb640 1 -- 172.21.15.98:0/2983600879 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3795385423 0 0) 0x7f2a9814aae0 con 0x7f2a9814ff40 2024-03-29T11:56:40.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.330+0000 7f2a8dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:56:40.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.330+0000 7f2a8dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:56:40.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.330+0000 7f2a8dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:56:40.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.330+0000 7f2a8dffb640 10 monclient(hunting): my global_id is 4515 2024-03-29T11:56:40.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.330+0000 7f2a8dffb640 1 -- 172.21.15.98:0/2983600879 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a98144990 con 0x7f2a9814ff40 2024-03-29T11:56:40.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.331+0000 7f2a8dffb640 1 -- 172.21.15.98:0/2983600879 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3253180310 0 0) 0x7f2a98145480 con 0x7f2a980a9930 2024-03-29T11:56:40.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.331+0000 7f2a8dffb640 1 -- 172.21.15.98:0/2983600879 >> v1:172.21.15.98:6790/0 conn(0x7f2a98149dc0 legacy=0x7f2a98124230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:40.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.331+0000 7f2a8dffb640 1 -- 172.21.15.98:0/2983600879 >> v1:172.21.15.98:6789/0 conn(0x7f2a9814ff40 legacy=0x7f2a98127960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:40.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.331+0000 7f2a8dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:56:40.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.331+0000 7f2a8dffb640 1 monclient: found mon.b 2024-03-29T11:56:40.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.331+0000 7f2a8dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:56:40.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.331+0000 7f2a8dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:56:40.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.331+0000 7f2a8dffb640 1 -- 172.21.15.98:0/2983600879 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a9812f800 con 0x7f2a980a9930 2024-03-29T11:56:40.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.331+0000 7f2a8dffb640 10 monclient: _finish_auth 0 2024-03-29T11:56:40.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.331+0000 7f2a8dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:56:40.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.331+0000 7f2a8dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:56:40.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.331+0000 7f2a9f406640 5 monclient: authenticate success, global_id 4516 2024-03-29T11:56:40.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.331+0000 7f2a9f406640 10 monclient: _renew_subs 2024-03-29T11:56:40.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.331+0000 7f2a9f406640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:56:40.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.331+0000 7f2a9f406640 1 -- 172.21.15.98:0/2983600879 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a981318a0 con 0x7f2a980a9930 2024-03-29T11:56:40.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.331+0000 7f2a9f406640 10 client.4516.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:56:40.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.331+0000 7f2a9f406640 10 monclient: _renew_subs 2024-03-29T11:56:40.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.331+0000 7f2a9f406640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:56:40.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.331+0000 7f2a9f406640 1 -- 172.21.15.98:0/2983600879 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a98130990 con 0x7f2a980a9930 2024-03-29T11:56:40.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.331+0000 7f2a9f406640 1 librados: init done 2024-03-29T11:56:40.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.331+0000 7f2a8dffb640 1 -- 172.21.15.98:0/2983600879 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a88015880 con 0x7f2a980a9930 2024-03-29T11:56:40.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.331+0000 7f2a8dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:56:40.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.331+0000 7f2a8dffb640 1 -- 172.21.15.98:0/2983600879 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f2a88015c90 con 0x7f2a980a9930 2024-03-29T11:56:40.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.331+0000 7f2a8dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:56:40.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.331+0000 7f2a8dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:56:40.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.331+0000 7f2a8dffb640 10 monclient: dump: 2024-03-29T11:56:40.344 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:56:40.344 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:56:40.344 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:40.344 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:40.344 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:56:40.344 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:40.345 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:56:40.345 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:56:40.345 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:56:40.345 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:40.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.333+0000 7f2a8dffb640 1 -- 172.21.15.98:0/2983600879 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f2a88015df0 con 0x7f2a980a9930 2024-03-29T11:56:40.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.334+0000 7f2a9f406640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:56:40.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.334+0000 7f2a9f406640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:56:40.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.334+0000 7f2a9f406640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:56:40.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.334+0000 7f2a9f406640 1 -- 172.21.15.98:0/2983600879 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a9812f9b0 con 0x7f2a980a9930 2024-03-29T11:56:40.346 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.334+0000 7f2a8dffb640 1 -- 172.21.15.98:0/2983600879 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f2a88095ae0 con 0x7f2a980a9930 2024-03-29T11:56:40.346 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.334+0000 7f2a8dffb640 10 client.4516.objecter ms_dispatch 0x7f2a98000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:56:40.346 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.334+0000 7f2a8f7fe640 4 set_mon_vals no callback set 2024-03-29T11:56:40.346 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.334+0000 7f2a8dffb640 3 client.4516.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:56:40.346 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.334+0000 7f2a8dffb640 3 client.4516.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:56:40.346 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.334+0000 7f2a8dffb640 20 client.4516.objecter dump_active .. 0 homeless 2024-03-29T11:56:40.346 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.337+0000 7f2a9c9a1640 10 monclient: get_auth_request con 0x7f2a7c06ddc0 auth_method 0 2024-03-29T11:56:40.346 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.337+0000 7f2a8dffb640 10 client.4516.objecter ms_handle_connect 0x7f2a7c06ddc0 2024-03-29T11:56:40.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.341+0000 7f2a8dffb640 1 -- 172.21.15.98:0/2983600879 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2a9812f9b0 con 0x7f2a980a9930 2024-03-29T11:56:40.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.341+0000 7f2a8dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:56:40.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.341+0000 7f2a8dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:56:40.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.666+0000 7f2a9f406640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:56:40.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.666+0000 7f2a9f406640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:56:40.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.666+0000 7f2a9f406640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:56:40.678 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.666+0000 7f2a9f406640 1 -- 172.21.15.98:0/2983600879 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2a9812f9b0 con 0x7f2a980a9930 2024-03-29T11:56:40.678 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.668+0000 7f2a8dffb640 1 -- 172.21.15.98:0/2983600879 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f2a9812f9b0 con 0x7f2a980a9930 2024-03-29T11:56:40.678 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.668+0000 7f2a8dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:56:40.678 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.668+0000 7f2a8dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:56:40.678 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:56:40.678 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:56:40.680 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.672+0000 7f2a9f406640 10 librados: watch_flush enter 2024-03-29T11:56:40.680 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.672+0000 7f2a9f406640 10 librados: watch_flush exit 2024-03-29T11:56:40.680 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.672+0000 7f2a9f406640 20 client.4516.objecter shutdown clearing up homeless session... 2024-03-29T11:56:40.680 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.672+0000 7f2a9f406640 10 client.4516.objecter successfully canceled tick 2024-03-29T11:56:40.681 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.672+0000 7f2a9f406640 1 -- 172.21.15.98:0/2983600879 >> v1:172.21.15.98:6816/4065186974 conn(0x7f2a7c06ddc0 legacy=0x7f2a7c070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:40.681 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.672+0000 7f2a9f406640 10 monclient: shutdown 2024-03-29T11:56:40.681 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.672+0000 7f2a9f406640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:56:40.681 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.672+0000 7f2a9f406640 1 -- 172.21.15.98:0/2983600879 >> v1:172.21.15.121:6789/0 conn(0x7f2a980a9930 legacy=0x7f2a98130280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:40.681 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.673+0000 7f2a9f406640 1 -- 172.21.15.98:0/2983600879 shutdown_connections 2024-03-29T11:56:40.681 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.673+0000 7f2a9f406640 1 -- 172.21.15.98:0/2983600879 >> 172.21.15.98:0/2983600879 conn(0x7f2a980b35e0 msgr2=0x7f2a980b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:56:40.682 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.673+0000 7f2a9f406640 1 -- 172.21.15.98:0/2983600879 shutdown_connections 2024-03-29T11:56:40.685 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.677+0000 7f2a9f406640 1 -- 172.21.15.98:0/2983600879 wait complete. 2024-03-29T11:56:40.685 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:40.677+0000 7f2a9f406640 1 librados: shutdown 2024-03-29T11:56:40.702 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:56:47.510 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:56:47.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.670+0000 7fc90759e640 10 monclient: get_monmap_and_config 2024-03-29T11:56:47.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.670+0000 7fc90759e640 10 monclient: build_initial_monmap 2024-03-29T11:56:47.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.670+0000 7fc90759e640 10 monclient: monmap: 2024-03-29T11:56:47.680 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:56:47.680 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:56:47.680 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:56:47.671851+0000 2024-03-29T11:56:47.680 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:56:47.671851+0000 2024-03-29T11:56:47.680 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:56:47.680 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:47.680 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:56:47.680 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:56:47.680 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:56:47.680 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:47.681 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.672+0000 7fc90759e640 1 Processor -- start 2024-03-29T11:56:47.681 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.672+0000 7fc90759e640 1 -- start start 2024-03-29T11:56:47.681 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.672+0000 7fc90759e640 10 monclient: init 2024-03-29T11:56:47.681 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.673+0000 7fc90759e640 10 monclient: _reopen_session rank -1 2024-03-29T11:56:47.683 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.673+0000 7fc90759e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:56:47.683 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.673+0000 7fc90759e640 10 monclient(hunting): picked mon.noname-c con 0x7fc900006800 addr v1:172.21.15.121:6789/0 2024-03-29T11:56:47.683 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.673+0000 7fc90759e640 10 monclient(hunting): picked mon.noname-a con 0x7fc900006be0 addr v1:172.21.15.98:6789/0 2024-03-29T11:56:47.683 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.673+0000 7fc90759e640 10 monclient(hunting): picked mon.noname-b con 0x7fc900031360 addr v1:172.21.15.98:6790/0 2024-03-29T11:56:47.683 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.673+0000 7fc90759e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc900031fa0 con 0x7fc900006be0 2024-03-29T11:56:47.683 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.673+0000 7fc90759e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9000295c0 con 0x7fc900006800 2024-03-29T11:56:47.683 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.673+0000 7fc90759e640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc90002c050 con 0x7fc900031360 2024-03-29T11:56:47.683 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.673+0000 7fc90759e640 10 monclient(hunting): _renew_subs 2024-03-29T11:56:47.683 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.673+0000 7fc90759e640 10 monclient(hunting): authenticate will time out at 1314.758363s 2024-03-29T11:56:47.686 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.673+0000 7fc906d9d640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fc900031360 0x7fc9000d30c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:53584/0 (socket says 172.21.15.98:53584) 2024-03-29T11:56:47.686 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.673+0000 7fc906d9d640 1 -- 172.21.15.98:0/2991206207 learned_addr learned my addr 172.21.15.98:0/2991206207 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:56:47.686 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.676+0000 7fc90559a640 1 -- 172.21.15.98:0/2991206207 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1900038556 0 0) 0x7fc90002c050 con 0x7fc900031360 2024-03-29T11:56:47.686 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.676+0000 7fc90559a640 10 monclient(hunting): _negotiate 2024-03-29T11:56:47.686 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.676+0000 7fc90559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:56:47.686 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.676+0000 7fc90559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:56:47.686 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.676+0000 7fc90559a640 10 monclient(hunting): my global_id is 4550 2024-03-29T11:56:47.686 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.676+0000 7fc90559a640 1 -- 172.21.15.98:0/2991206207 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8f4003520 con 0x7fc900031360 2024-03-29T11:56:47.686 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.677+0000 7fc90559a640 1 -- 172.21.15.98:0/2991206207 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1022605487 0 0) 0x7fc8f4003520 con 0x7fc900031360 2024-03-29T11:56:47.686 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.677+0000 7fc90559a640 1 -- 172.21.15.98:0/2991206207 >> v1:172.21.15.121:6789/0 conn(0x7fc900006800 legacy=0x7fc9000cad20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:56:47.686 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.677+0000 7fc90559a640 1 -- 172.21.15.98:0/2991206207 >> v1:172.21.15.98:6789/0 conn(0x7fc900006be0 legacy=0x7fc9000c4ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:56:47.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.679+0000 7fc90559a640 10 monclient: _finish_hunting 0 2024-03-29T11:56:47.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.679+0000 7fc90559a640 1 monclient: found mon.noname-b 2024-03-29T11:56:47.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.679+0000 7fc90559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:56:47.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.679+0000 7fc90559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:56:47.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.679+0000 7fc90559a640 1 -- 172.21.15.98:0/2991206207 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9000cb430 con 0x7fc900031360 2024-03-29T11:56:47.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.679+0000 7fc90559a640 10 monclient: _finish_auth 0 2024-03-29T11:56:47.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.679+0000 7fc90559a640 10 monclient: _check_auth_tickets 2024-03-29T11:56:47.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.679+0000 7fc90559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:56:47.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.679+0000 7fc90559a640 1 -- 172.21.15.98:0/2991206207 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fc90804f730 con 0x7fc900031360 2024-03-29T11:56:47.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.679+0000 7fc90559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:56:47.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.679+0000 7fc90559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:56:47.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.679+0000 7fc90559a640 10 monclient: dump: 2024-03-29T11:56:47.690 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:56:47.690 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:56:47.690 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:47.690 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:47.690 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:56:47.690 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:47.690 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:56:47.690 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:56:47.690 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:56:47.690 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:47.690 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.679+0000 7fc90759e640 5 monclient: authenticate success, global_id 4550 2024-03-29T11:56:47.690 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.679+0000 7fc90759e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:56:47.690 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.680+0000 7fc90559a640 1 -- 172.21.15.98:0/2991206207 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc908075f10 con 0x7fc900031360 2024-03-29T11:56:47.690 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.680+0000 7fc90559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:56:47.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.680+0000 7fc90559a640 1 -- 172.21.15.98:0/2991206207 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fc9080758b0 con 0x7fc900031360 2024-03-29T11:56:47.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.680+0000 7fc90559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:56:47.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.680+0000 7fc90559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:56:47.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.680+0000 7fc90559a640 10 monclient: dump: 2024-03-29T11:56:47.691 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:56:47.691 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:56:47.691 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:47.691 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:47.691 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:56:47.691 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:47.691 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:56:47.691 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:56:47.691 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:56:47.691 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:47.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.684+0000 7fc90759e640 10 monclient: get_monmap_and_config success 2024-03-29T11:56:47.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.684+0000 7fc90759e640 4 set_mon_vals no callback set 2024-03-29T11:56:47.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.685+0000 7fc90759e640 10 monclient: shutdown 2024-03-29T11:56:47.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.685+0000 7fc90759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:56:47.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.685+0000 7fc90759e640 1 -- 172.21.15.98:0/2991206207 >> v1:172.21.15.98:6790/0 conn(0x7fc900031360 legacy=0x7fc9000d30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:47.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.685+0000 7fc90759e640 1 -- 172.21.15.98:0/2991206207 shutdown_connections 2024-03-29T11:56:47.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.685+0000 7fc90759e640 1 -- 172.21.15.98:0/2991206207 >> 172.21.15.98:0/2991206207 conn(0x7fc9000058f0 msgr2=0x7fc900005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:56:47.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.686+0000 7fc90759e640 1 -- 172.21.15.98:0/2991206207 shutdown_connections 2024-03-29T11:56:47.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.686+0000 7fc90759e640 1 -- 172.21.15.98:0/2991206207 wait complete. 2024-03-29T11:56:47.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.687+0000 7fc90759e640 10 monclient: build_initial_monmap 2024-03-29T11:56:47.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.687+0000 7fc90759e640 10 monclient: monmap: 2024-03-29T11:56:47.696 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:56:47.696 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:56:47.696 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:56:47.696 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:56:47.696 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:56:47.697 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:47.697 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:56:47.697 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:56:47.697 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:56:47.697 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:47.697 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.687+0000 7fc90759e640 1 librados: starting msgr at 2024-03-29T11:56:47.697 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.687+0000 7fc90759e640 1 librados: starting objecter 2024-03-29T11:56:47.697 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.687+0000 7fc90759e640 1 Processor -- start 2024-03-29T11:56:47.699 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.688+0000 7fc90759e640 1 -- start start 2024-03-29T11:56:47.699 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.688+0000 7fc90759e640 1 librados: setting wanted keys 2024-03-29T11:56:47.699 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.688+0000 7fc90759e640 1 librados: calling monclient init 2024-03-29T11:56:47.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.688+0000 7fc90759e640 10 monclient: init 2024-03-29T11:56:47.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.689+0000 7fc90759e640 10 monclient: _reopen_session rank -1 2024-03-29T11:56:47.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.689+0000 7fc90759e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:56:47.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.689+0000 7fc90759e640 10 monclient(hunting): picked mon.noname-c con 0x7fc900006800 addr v1:172.21.15.98:6790/0 2024-03-29T11:56:47.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.689+0000 7fc90759e640 10 monclient(hunting): picked mon.noname-b con 0x7fc900031360 addr v1:172.21.15.121:6789/0 2024-03-29T11:56:47.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.689+0000 7fc90759e640 10 monclient(hunting): picked mon.noname-a con 0x7fc9000c1620 addr v1:172.21.15.98:6789/0 2024-03-29T11:56:47.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.689+0000 7fc90759e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc900030000 con 0x7fc9000c1620 2024-03-29T11:56:47.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.689+0000 7fc90759e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc900164090 con 0x7fc900031360 2024-03-29T11:56:47.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.689+0000 7fc90759e640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9000c0b50 con 0x7fc900006800 2024-03-29T11:56:47.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.689+0000 7fc90759e640 10 monclient(hunting): _renew_subs 2024-03-29T11:56:47.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.689+0000 7fc90759e640 10 monclient(hunting): authenticate will time out at 1314.774203s 2024-03-29T11:56:47.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.690+0000 7fc90659c640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fc900006800 0x7fc9000c0440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:54746/0 (socket says 172.21.15.98:54746) 2024-03-29T11:56:47.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.690+0000 7fc90659c640 1 -- 172.21.15.98:0/1652696911 learned_addr learned my addr 172.21.15.98:0/1652696911 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:56:47.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.691+0000 7fc8feffd640 10 client.?.objecter ms_handle_connect 0x7fc900006800 2024-03-29T11:56:47.701 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.691+0000 7fc8feffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:56:47.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.692+0000 7fc8feffd640 1 -- 172.21.15.98:0/1652696911 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3994428251 0 0) 0x7fc9000c0b50 con 0x7fc900006800 2024-03-29T11:56:47.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.692+0000 7fc8feffd640 10 monclient(hunting): _negotiate 2024-03-29T11:56:47.703 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.692+0000 7fc8feffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:56:47.703 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.692+0000 7fc8feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:56:47.703 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.692+0000 7fc8feffd640 10 monclient(hunting): my global_id is 4553 2024-03-29T11:56:47.703 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.692+0000 7fc8feffd640 1 -- 172.21.15.98:0/1652696911 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8e8004eb0 con 0x7fc900006800 2024-03-29T11:56:47.705 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.694+0000 7fc8feffd640 1 -- 172.21.15.98:0/1652696911 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3396607746 0 0) 0x7fc8e8004eb0 con 0x7fc900006800 2024-03-29T11:56:47.705 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.694+0000 7fc8feffd640 1 -- 172.21.15.98:0/1652696911 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9000c0b50 con 0x7fc900006800 2024-03-29T11:56:47.705 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.694+0000 7fc8feffd640 1 -- 172.21.15.98:0/1652696911 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fc8f80017d0 con 0x7fc900006800 2024-03-29T11:56:47.705 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.694+0000 7fc8feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:56:47.705 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.694+0000 7fc8feffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:56:47.705 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.694+0000 7fc8feffd640 10 monclient(hunting): dump: 2024-03-29T11:56:47.705 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:56:47.705 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:56:47.705 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:47.705 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:47.705 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:56:47.705 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:47.705 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:56:47.706 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:56:47.706 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:56:47.706 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:47.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.694+0000 7fc8feffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:56:47.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.695+0000 7fc8feffd640 10 client.?.objecter ms_handle_connect 0x7fc900031360 2024-03-29T11:56:47.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.695+0000 7fc8feffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:56:47.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.695+0000 7fc8feffd640 10 client.?.objecter ms_handle_connect 0x7fc9000c1620 2024-03-29T11:56:47.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.695+0000 7fc8feffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:56:47.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.695+0000 7fc8feffd640 1 -- 172.21.15.98:0/1652696911 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3435782642 0 0) 0x7fc900164090 con 0x7fc900031360 2024-03-29T11:56:47.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.695+0000 7fc8feffd640 10 monclient(hunting): _negotiate 2024-03-29T11:56:47.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.695+0000 7fc8feffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:56:47.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.695+0000 7fc8feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:56:47.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.695+0000 7fc8feffd640 10 monclient(hunting): my global_id is 4519 2024-03-29T11:56:47.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.695+0000 7fc8feffd640 1 -- 172.21.15.98:0/1652696911 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8e8004eb0 con 0x7fc900031360 2024-03-29T11:56:47.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.699+0000 7fc8feffd640 1 -- 172.21.15.98:0/1652696911 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1182054787 0 0) 0x7fc900030000 con 0x7fc9000c1620 2024-03-29T11:56:47.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.699+0000 7fc8feffd640 10 monclient(hunting): _negotiate 2024-03-29T11:56:47.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.699+0000 7fc8feffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:56:47.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.699+0000 7fc8feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:56:47.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.699+0000 7fc8feffd640 10 monclient(hunting): my global_id is 4518 2024-03-29T11:56:47.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.699+0000 7fc8feffd640 1 -- 172.21.15.98:0/1652696911 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc900164090 con 0x7fc9000c1620 2024-03-29T11:56:47.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.699+0000 7fc8feffd640 1 -- 172.21.15.98:0/1652696911 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3273938871 0 0) 0x7fc9000c0b50 con 0x7fc900006800 2024-03-29T11:56:47.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.699+0000 7fc8feffd640 1 -- 172.21.15.98:0/1652696911 >> v1:172.21.15.121:6789/0 conn(0x7fc900031360 legacy=0x7fc90002e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:47.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.699+0000 7fc8feffd640 1 -- 172.21.15.98:0/1652696911 >> v1:172.21.15.98:6789/0 conn(0x7fc9000c1620 legacy=0x7fc90016c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:47.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.699+0000 7fc8feffd640 10 monclient: _finish_hunting 0 2024-03-29T11:56:47.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.699+0000 7fc8feffd640 1 monclient: found mon.c 2024-03-29T11:56:47.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.699+0000 7fc8feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:56:47.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.699+0000 7fc8feffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:56:47.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.699+0000 7fc8feffd640 1 -- 172.21.15.98:0/1652696911 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9000c0ce0 con 0x7fc900006800 2024-03-29T11:56:47.709 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.699+0000 7fc8feffd640 10 monclient: _finish_auth 0 2024-03-29T11:56:47.709 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.699+0000 7fc8feffd640 10 monclient: _check_auth_tickets 2024-03-29T11:56:47.709 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.699+0000 7fc8feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:56:47.709 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.699+0000 7fc90759e640 5 monclient: authenticate success, global_id 4553 2024-03-29T11:56:47.709 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.699+0000 7fc90759e640 10 monclient: _renew_subs 2024-03-29T11:56:47.709 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.699+0000 7fc90759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:56:47.709 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.699+0000 7fc90759e640 1 -- 172.21.15.98:0/1652696911 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9000c7980 con 0x7fc900006800 2024-03-29T11:56:47.709 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.699+0000 7fc90759e640 10 client.4553.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:56:47.709 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.699+0000 7fc90759e640 10 monclient: _renew_subs 2024-03-29T11:56:47.709 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.699+0000 7fc90759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:56:47.709 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.699+0000 7fc90759e640 1 -- 172.21.15.98:0/1652696911 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc90002dd60 con 0x7fc900006800 2024-03-29T11:56:47.709 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.699+0000 7fc90759e640 1 librados: init done 2024-03-29T11:56:47.710 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.700+0000 7fc90759e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:56:47.710 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.700+0000 7fc90759e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:56:47.710 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.701+0000 7fc90759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:56:47.710 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.701+0000 7fc90759e640 1 -- 172.21.15.98:0/1652696911 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc900032410 con 0x7fc900006800 2024-03-29T11:56:47.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.703+0000 7fc8feffd640 1 -- 172.21.15.98:0/1652696911 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8f8001930 con 0x7fc900006800 2024-03-29T11:56:47.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.703+0000 7fc8feffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:56:47.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.703+0000 7fc904d99640 4 set_mon_vals no callback set 2024-03-29T11:56:47.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.704+0000 7fc8feffd640 1 -- 172.21.15.98:0/1652696911 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fc8f8004a00 con 0x7fc900006800 2024-03-29T11:56:47.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.704+0000 7fc8feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:56:47.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.704+0000 7fc8feffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:56:47.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.704+0000 7fc8feffd640 10 monclient: dump: 2024-03-29T11:56:47.713 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:56:47.713 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:56:47.713 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:47.713 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:47.713 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:56:47.713 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:47.713 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:56:47.713 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:56:47.713 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:56:47.713 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:47.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.704+0000 7fc8feffd640 1 -- 172.21.15.98:0/1652696911 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7fc8f801a9d0 con 0x7fc900006800 2024-03-29T11:56:47.717 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.705+0000 7fc905d9b640 10 monclient: get_auth_request con 0x7fc8e806daa0 auth_method 0 2024-03-29T11:56:47.717 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.705+0000 7fc8feffd640 10 client.4553.objecter ms_handle_connect 0x7fc8e806daa0 2024-03-29T11:56:47.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.705+0000 7fc8feffd640 1 -- 172.21.15.98:0/1652696911 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7fc8f8084ae0 con 0x7fc900006800 2024-03-29T11:56:47.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.705+0000 7fc8feffd640 10 client.4553.objecter ms_dispatch 0x7fc908000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:56:47.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.705+0000 7fc8feffd640 3 client.4553.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:56:47.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.705+0000 7fc8feffd640 3 client.4553.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:56:47.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.705+0000 7fc8feffd640 20 client.4553.objecter dump_active .. 0 homeless 2024-03-29T11:56:47.721 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.713+0000 7fc8feffd640 1 -- 172.21.15.98:0/1652696911 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc900032410 con 0x7fc900006800 2024-03-29T11:56:47.721 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.713+0000 7fc8feffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:56:47.721 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:47.713+0000 7fc8feffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:56:48.082 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:48.074+0000 7fc90759e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:56:48.082 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:48.074+0000 7fc90759e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:56:48.082 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:48.074+0000 7fc90759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:56:48.082 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:48.074+0000 7fc90759e640 1 -- 172.21.15.98:0/1652696911 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc900032410 con 0x7fc900006800 2024-03-29T11:56:48.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:48.075+0000 7fc8feffd640 1 -- 172.21.15.98:0/1652696911 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7fc900032410 con 0x7fc900006800 2024-03-29T11:56:48.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:48.075+0000 7fc8feffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:56:48.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:48.075+0000 7fc8feffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:56:48.084 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:56:48.084 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:56:48.087 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:48.078+0000 7fc90759e640 10 librados: watch_flush enter 2024-03-29T11:56:48.087 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:48.078+0000 7fc90759e640 10 librados: watch_flush exit 2024-03-29T11:56:48.087 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:48.078+0000 7fc90759e640 20 client.4553.objecter shutdown clearing up homeless session... 2024-03-29T11:56:48.087 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:48.078+0000 7fc90759e640 10 client.4553.objecter successfully canceled tick 2024-03-29T11:56:48.087 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:48.079+0000 7fc90759e640 1 -- 172.21.15.98:0/1652696911 >> v1:172.21.15.98:6816/4065186974 conn(0x7fc8e806daa0 legacy=0x7fc8e806ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:48.087 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:48.079+0000 7fc90759e640 10 monclient: shutdown 2024-03-29T11:56:48.087 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:48.079+0000 7fc90759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:56:48.087 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:48.079+0000 7fc90759e640 1 -- 172.21.15.98:0/1652696911 >> v1:172.21.15.98:6790/0 conn(0x7fc900006800 legacy=0x7fc9000c0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:48.087 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:48.079+0000 7fc90759e640 1 -- 172.21.15.98:0/1652696911 shutdown_connections 2024-03-29T11:56:48.088 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:48.079+0000 7fc90759e640 1 -- 172.21.15.98:0/1652696911 >> 172.21.15.98:0/1652696911 conn(0x7fc9000058f0 msgr2=0x7fc9000334d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:56:48.088 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:48.080+0000 7fc90759e640 1 -- 172.21.15.98:0/1652696911 shutdown_connections 2024-03-29T11:56:48.088 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:48.080+0000 7fc90759e640 1 -- 172.21.15.98:0/1652696911 wait complete. 2024-03-29T11:56:48.089 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:48.080+0000 7fc90759e640 1 librados: shutdown 2024-03-29T11:56:48.110 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:56:54.918 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:56:55.077 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.068+0000 7fd5028e2640 10 monclient: get_monmap_and_config 2024-03-29T11:56:55.077 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.068+0000 7fd5028e2640 10 monclient: build_initial_monmap 2024-03-29T11:56:55.077 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.068+0000 7fd5028e2640 10 monclient: monmap: 2024-03-29T11:56:55.077 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:56:55.078 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:56:55.078 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:56:55.070031+0000 2024-03-29T11:56:55.078 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:56:55.070031+0000 2024-03-29T11:56:55.078 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:56:55.078 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:55.078 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:56:55.078 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:56:55.078 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:56:55.078 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:55.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.069+0000 7fd5028e2640 1 Processor -- start 2024-03-29T11:56:55.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.069+0000 7fd5028e2640 1 -- start start 2024-03-29T11:56:55.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.069+0000 7fd5028e2640 10 monclient: init 2024-03-29T11:56:55.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.070+0000 7fd5028e2640 10 monclient: _reopen_session rank -1 2024-03-29T11:56:55.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.070+0000 7fd5028e2640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:56:55.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.070+0000 7fd5028e2640 10 monclient(hunting): picked mon.noname-a con 0x7fd4fc1457f0 addr v1:172.21.15.98:6789/0 2024-03-29T11:56:55.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.070+0000 7fd5028e2640 10 monclient(hunting): picked mon.noname-b con 0x7fd4fc13f650 addr v1:172.21.15.98:6790/0 2024-03-29T11:56:55.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.070+0000 7fd5028e2640 10 monclient(hunting): picked mon.noname-c con 0x7fd4fc0a9d70 addr v1:172.21.15.121:6789/0 2024-03-29T11:56:55.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.070+0000 7fd5028e2640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4fc0aaef0 con 0x7fd4fc1457f0 2024-03-29T11:56:55.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.070+0000 7fd5028e2640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4fc145bd0 con 0x7fd4fc0a9d70 2024-03-29T11:56:55.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.070+0000 7fd5028e2640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4fc145eb0 con 0x7fd4fc13f650 2024-03-29T11:56:55.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.070+0000 7fd5028e2640 10 monclient(hunting): _renew_subs 2024-03-29T11:56:55.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.070+0000 7fd5028e2640 10 monclient(hunting): authenticate will time out at 1322.156500s 2024-03-29T11:56:55.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.070+0000 7fd4fb7fe640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fd4fc13f650 0x7fd4fc0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:43666/0 (socket says 172.21.15.98:43666) 2024-03-29T11:56:55.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.070+0000 7fd4fb7fe640 1 -- 172.21.15.98:0/645915200 learned_addr learned my addr 172.21.15.98:0/645915200 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:56:55.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.071+0000 7fd4faffd640 1 -- 172.21.15.98:0/645915200 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3367173539 0 0) 0x7fd4fc145bd0 con 0x7fd4fc0a9d70 2024-03-29T11:56:55.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.071+0000 7fd4faffd640 10 monclient(hunting): _negotiate 2024-03-29T11:56:55.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.071+0000 7fd4faffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:56:55.082 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.071+0000 7fd4faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:56:55.082 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.071+0000 7fd4faffd640 10 monclient(hunting): my global_id is 4522 2024-03-29T11:56:55.082 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.071+0000 7fd4faffd640 1 -- 172.21.15.98:0/645915200 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4e4003520 con 0x7fd4fc0a9d70 2024-03-29T11:56:55.082 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.071+0000 7fd4faffd640 1 -- 172.21.15.98:0/645915200 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2558698988 0 0) 0x7fd4fc0aaef0 con 0x7fd4fc1457f0 2024-03-29T11:56:55.082 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.071+0000 7fd4faffd640 10 monclient(hunting): _negotiate 2024-03-29T11:56:55.082 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.071+0000 7fd4faffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:56:55.082 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.071+0000 7fd4faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:56:55.082 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.071+0000 7fd4faffd640 10 monclient(hunting): my global_id is 4521 2024-03-29T11:56:55.082 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.071+0000 7fd4faffd640 1 -- 172.21.15.98:0/645915200 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4fc145bd0 con 0x7fd4fc1457f0 2024-03-29T11:56:55.082 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.072+0000 7fd4faffd640 1 -- 172.21.15.98:0/645915200 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1719214532 0 0) 0x7fd4e4003520 con 0x7fd4fc0a9d70 2024-03-29T11:56:55.082 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.072+0000 7fd4faffd640 1 -- 172.21.15.98:0/645915200 >> v1:172.21.15.98:6790/0 conn(0x7fd4fc13f650 legacy=0x7fd4fc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:55.082 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.072+0000 7fd4faffd640 1 -- 172.21.15.98:0/645915200 >> v1:172.21.15.98:6789/0 conn(0x7fd4fc1457f0 legacy=0x7fd4fc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:55.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.072+0000 7fd4faffd640 10 monclient: _finish_hunting 0 2024-03-29T11:56:55.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.072+0000 7fd4faffd640 1 monclient: found mon.noname-c 2024-03-29T11:56:55.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.072+0000 7fd4faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:56:55.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.072+0000 7fd4faffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:56:55.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.072+0000 7fd4faffd640 1 -- 172.21.15.98:0/645915200 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4fc146340 con 0x7fd4fc0a9d70 2024-03-29T11:56:55.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.072+0000 7fd4faffd640 10 monclient: _finish_auth 0 2024-03-29T11:56:55.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.072+0000 7fd4faffd640 10 monclient: _check_auth_tickets 2024-03-29T11:56:55.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.072+0000 7fd4faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:56:55.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.072+0000 7fd4faffd640 1 -- 172.21.15.98:0/645915200 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fd4f4001690 con 0x7fd4fc0a9d70 2024-03-29T11:56:55.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.072+0000 7fd4faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:56:55.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.072+0000 7fd4faffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:56:55.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.072+0000 7fd4faffd640 10 monclient: dump: 2024-03-29T11:56:55.083 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:56:55.083 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:56:55.083 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:55.083 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:55.084 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:56:55.084 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:55.084 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:56:55.084 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:56:55.084 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:56:55.084 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:55.084 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.072+0000 7fd5028e2640 5 monclient: authenticate success, global_id 4522 2024-03-29T11:56:55.084 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.072+0000 7fd5028e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:56:55.086 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.073+0000 7fd4faffd640 1 -- 172.21.15.98:0/645915200 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd4f40017f0 con 0x7fd4fc0a9d70 2024-03-29T11:56:55.086 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.073+0000 7fd4faffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:56:55.086 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.073+0000 7fd4faffd640 1 -- 172.21.15.98:0/645915200 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fd4f4001c40 con 0x7fd4fc0a9d70 2024-03-29T11:56:55.087 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.073+0000 7fd4faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:56:55.087 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.073+0000 7fd4faffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:56:55.087 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.073+0000 7fd4faffd640 10 monclient: dump: 2024-03-29T11:56:55.087 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:56:55.087 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:56:55.087 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:55.087 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:55.087 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:56:55.087 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:55.087 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:56:55.087 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:56:55.087 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:56:55.087 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:55.087 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.073+0000 7fd5028e2640 10 monclient: get_monmap_and_config success 2024-03-29T11:56:55.087 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.073+0000 7fd5028e2640 4 set_mon_vals no callback set 2024-03-29T11:56:55.088 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.073+0000 7fd5028e2640 10 monclient: shutdown 2024-03-29T11:56:55.088 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.073+0000 7fd5028e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:56:55.088 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.073+0000 7fd5028e2640 1 -- 172.21.15.98:0/645915200 >> v1:172.21.15.121:6789/0 conn(0x7fd4fc0a9d70 legacy=0x7fd4fc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:55.088 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.074+0000 7fd5028e2640 1 -- 172.21.15.98:0/645915200 shutdown_connections 2024-03-29T11:56:55.088 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.074+0000 7fd5028e2640 1 -- 172.21.15.98:0/645915200 >> 172.21.15.98:0/645915200 conn(0x7fd4fc066a00 msgr2=0x7fd4fc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:56:55.088 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.074+0000 7fd5028e2640 1 -- 172.21.15.98:0/645915200 shutdown_connections 2024-03-29T11:56:55.088 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.074+0000 7fd5028e2640 1 -- 172.21.15.98:0/645915200 wait complete. 2024-03-29T11:56:55.088 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.075+0000 7fd5028e2640 10 monclient: build_initial_monmap 2024-03-29T11:56:55.088 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.075+0000 7fd5028e2640 10 monclient: monmap: 2024-03-29T11:56:55.089 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:56:55.089 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:56:55.089 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:56:55.089 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:56:55.089 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:56:55.089 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:55.089 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:56:55.089 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:56:55.089 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:56:55.089 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:55.089 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.075+0000 7fd5028e2640 1 librados: starting msgr at 2024-03-29T11:56:55.089 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.075+0000 7fd5028e2640 1 librados: starting objecter 2024-03-29T11:56:55.089 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.075+0000 7fd5028e2640 1 Processor -- start 2024-03-29T11:56:55.089 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.076+0000 7fd5028e2640 1 -- start start 2024-03-29T11:56:55.089 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.076+0000 7fd5028e2640 1 librados: setting wanted keys 2024-03-29T11:56:55.090 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.076+0000 7fd5028e2640 1 librados: calling monclient init 2024-03-29T11:56:55.090 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.076+0000 7fd5028e2640 10 monclient: init 2024-03-29T11:56:55.090 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.077+0000 7fd5028e2640 10 monclient: _reopen_session rank -1 2024-03-29T11:56:55.090 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.077+0000 7fd5028e2640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:56:55.090 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.077+0000 7fd5028e2640 10 monclient(hunting): picked mon.noname-c con 0x7fd4fc0a9d70 addr v1:172.21.15.98:6790/0 2024-03-29T11:56:55.090 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.077+0000 7fd5028e2640 10 monclient(hunting): picked mon.noname-a con 0x7fd4fc13f650 addr v1:172.21.15.98:6789/0 2024-03-29T11:56:55.090 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.077+0000 7fd5028e2640 10 monclient(hunting): picked mon.noname-b con 0x7fd4fc1457f0 addr v1:172.21.15.121:6789/0 2024-03-29T11:56:55.090 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.077+0000 7fd5028e2640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4fc071ec0 con 0x7fd4fc13f650 2024-03-29T11:56:55.090 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.077+0000 7fd5028e2640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4fc13af50 con 0x7fd4fc1457f0 2024-03-29T11:56:55.090 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.077+0000 7fd4fbfff640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fd4fc0a9d70 0x7fd4fc1db6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:45646/0 (socket says 172.21.15.98:45646) 2024-03-29T11:56:55.090 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.077+0000 7fd5028e2640 1 -- 172.21.15.98:0/2781976342 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4fc1df130 con 0x7fd4fc0a9d70 2024-03-29T11:56:55.093 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.077+0000 7fd4fbfff640 1 -- 172.21.15.98:0/2781976342 learned_addr learned my addr 172.21.15.98:0/2781976342 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:56:55.094 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.077+0000 7fd5028e2640 10 monclient(hunting): _renew_subs 2024-03-29T11:56:55.094 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.077+0000 7fd5028e2640 10 monclient(hunting): authenticate will time out at 1322.163259s 2024-03-29T11:56:55.094 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.077+0000 7fd4f8ff9640 10 client.?.objecter ms_handle_connect 0x7fd4fc1457f0 2024-03-29T11:56:55.094 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.077+0000 7fd4f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:56:55.094 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.077+0000 7fd4f8ff9640 10 client.?.objecter ms_handle_connect 0x7fd4fc0a9d70 2024-03-29T11:56:55.094 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.077+0000 7fd4f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:56:55.099 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.078+0000 7fd4f8ff9640 1 -- 172.21.15.98:0/2781976342 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1068910579 0 0) 0x7fd4fc1df130 con 0x7fd4fc0a9d70 2024-03-29T11:56:55.099 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.078+0000 7fd4f8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:56:55.099 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.078+0000 7fd4f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:56:55.099 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.078+0000 7fd4f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:56:55.099 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.078+0000 7fd4f8ff9640 10 monclient(hunting): my global_id is 4559 2024-03-29T11:56:55.099 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.078+0000 7fd4f8ff9640 1 -- 172.21.15.98:0/2781976342 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4e8004eb0 con 0x7fd4fc0a9d70 2024-03-29T11:56:55.099 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.078+0000 7fd4f8ff9640 1 -- 172.21.15.98:0/2781976342 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3360708959 0 0) 0x7fd4fc13af50 con 0x7fd4fc1457f0 2024-03-29T11:56:55.099 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.078+0000 7fd4f8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:56:55.100 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.078+0000 7fd4f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:56:55.100 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.078+0000 7fd4f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:56:55.100 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.078+0000 7fd4f8ff9640 10 monclient(hunting): my global_id is 4525 2024-03-29T11:56:55.100 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.078+0000 7fd4f8ff9640 1 -- 172.21.15.98:0/2781976342 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4fc1df130 con 0x7fd4fc1457f0 2024-03-29T11:56:55.100 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.078+0000 7fd4f8ff9640 1 -- 172.21.15.98:0/2781976342 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3151531234 0 0) 0x7fd4e8004eb0 con 0x7fd4fc0a9d70 2024-03-29T11:56:55.100 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.079+0000 7fd4f8ff9640 1 -- 172.21.15.98:0/2781976342 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd4fc13af50 con 0x7fd4fc0a9d70 2024-03-29T11:56:55.100 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.079+0000 7fd4f8ff9640 1 -- 172.21.15.98:0/2781976342 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fd4f00018f0 con 0x7fd4fc0a9d70 2024-03-29T11:56:55.100 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.079+0000 7fd4f8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:56:55.100 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.079+0000 7fd4f8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:56:55.100 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.079+0000 7fd4f8ff9640 10 monclient(hunting): dump: 2024-03-29T11:56:55.100 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:56:55.100 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:56:55.100 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:55.100 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:55.100 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:56:55.101 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:55.101 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:56:55.101 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:56:55.101 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:56:55.101 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:55.101 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.079+0000 7fd4f8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:56:55.101 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.079+0000 7fd4f8ff9640 1 -- 172.21.15.98:0/2781976342 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2851178890 0 0) 0x7fd4fc1df130 con 0x7fd4fc1457f0 2024-03-29T11:56:55.101 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.079+0000 7fd4f8ff9640 1 -- 172.21.15.98:0/2781976342 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd4e8004eb0 con 0x7fd4fc1457f0 2024-03-29T11:56:55.101 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.079+0000 7fd4f8ff9640 1 -- 172.21.15.98:0/2781976342 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fd4f4001d80 con 0x7fd4fc1457f0 2024-03-29T11:56:55.101 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.079+0000 7fd4f8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:56:55.101 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.079+0000 7fd4f8ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:56:55.101 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.079+0000 7fd4f8ff9640 10 monclient(hunting): dump: 2024-03-29T11:56:55.101 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:56:55.101 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:56:55.101 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:55.102 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:55.102 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:56:55.102 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:55.102 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:56:55.102 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:56:55.102 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:56:55.102 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:55.102 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.079+0000 7fd4f8ff9640 10 client.?.objecter ms_handle_connect 0x7fd4fc13f650 2024-03-29T11:56:55.102 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.079+0000 7fd4f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:56:55.102 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.079+0000 7fd4f8ff9640 1 -- 172.21.15.98:0/2781976342 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1949041302 0 0) 0x7fd4fc13af50 con 0x7fd4fc0a9d70 2024-03-29T11:56:55.102 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.079+0000 7fd4f8ff9640 1 -- 172.21.15.98:0/2781976342 >> v1:172.21.15.121:6789/0 conn(0x7fd4fc1457f0 legacy=0x7fd4fc13a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:55.102 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.079+0000 7fd4f8ff9640 1 -- 172.21.15.98:0/2781976342 >> v1:172.21.15.98:6789/0 conn(0x7fd4fc13f650 legacy=0x7fd4fc13a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:55.102 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.079+0000 7fd4f8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:56:55.102 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.079+0000 7fd4f8ff9640 1 monclient: found mon.c 2024-03-29T11:56:55.103 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.079+0000 7fd4f8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:56:55.103 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.079+0000 7fd4f8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:56:55.103 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.080+0000 7fd4f8ff9640 1 -- 172.21.15.98:0/2781976342 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4fc1debc0 con 0x7fd4fc0a9d70 2024-03-29T11:56:55.103 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.080+0000 7fd4f8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:56:55.103 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.080+0000 7fd4f8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:56:55.103 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.080+0000 7fd4f8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:56:55.103 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.080+0000 7fd5028e2640 5 monclient: authenticate success, global_id 4559 2024-03-29T11:56:55.103 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.080+0000 7fd5028e2640 10 monclient: _renew_subs 2024-03-29T11:56:55.103 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.080+0000 7fd5028e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:56:55.103 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.080+0000 7fd5028e2640 1 -- 172.21.15.98:0/2781976342 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4fc1dedb0 con 0x7fd4fc0a9d70 2024-03-29T11:56:55.103 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.080+0000 7fd5028e2640 10 client.4559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:56:55.103 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.080+0000 7fd5028e2640 10 monclient: _renew_subs 2024-03-29T11:56:55.103 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.080+0000 7fd5028e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:56:55.103 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.080+0000 7fd5028e2640 1 -- 172.21.15.98:0/2781976342 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4fc0b0e10 con 0x7fd4fc0a9d70 2024-03-29T11:56:55.103 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.080+0000 7fd5028e2640 1 librados: init done 2024-03-29T11:56:55.104 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.080+0000 7fd4f8ff9640 1 -- 172.21.15.98:0/2781976342 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd4f0001a50 con 0x7fd4fc0a9d70 2024-03-29T11:56:55.104 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.080+0000 7fd4f8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:56:55.104 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.080+0000 7fd4f8ff9640 1 -- 172.21.15.98:0/2781976342 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fd4f0003790 con 0x7fd4fc0a9d70 2024-03-29T11:56:55.104 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.080+0000 7fd4f8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:56:55.104 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.080+0000 7fd4fa7fc640 4 set_mon_vals no callback set 2024-03-29T11:56:55.104 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.080+0000 7fd4f8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:56:55.104 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.080+0000 7fd4f8ff9640 10 monclient: dump: 2024-03-29T11:56:55.104 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:56:55.104 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:56:55.104 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:55.104 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:56:55.104 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:56:55.104 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:56:55.104 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:56:55.104 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:56:55.105 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:56:55.105 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:56:55.105 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.081+0000 7fd5028e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:56:55.105 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.081+0000 7fd5028e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:56:55.105 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.081+0000 7fd5028e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:56:55.105 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.081+0000 7fd5028e2640 1 -- 172.21.15.98:0/2781976342 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4dc005080 con 0x7fd4fc0a9d70 2024-03-29T11:56:55.105 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.083+0000 7fd4f8ff9640 1 -- 172.21.15.98:0/2781976342 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7fd4f00038f0 con 0x7fd4fc0a9d70 2024-03-29T11:56:55.105 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.084+0000 7fd4fb7fe640 10 monclient: get_auth_request con 0x7fd4e806dec0 auth_method 0 2024-03-29T11:56:55.105 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.084+0000 7fd4f8ff9640 1 -- 172.21.15.98:0/2781976342 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7fd4f00848e0 con 0x7fd4fc0a9d70 2024-03-29T11:56:55.105 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.084+0000 7fd4f8ff9640 10 client.4559.objecter ms_dispatch 0x7fd4fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:56:55.105 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.084+0000 7fd4f8ff9640 3 client.4559.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:56:55.105 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.084+0000 7fd4f8ff9640 3 client.4559.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:56:55.105 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.084+0000 7fd4f8ff9640 20 client.4559.objecter dump_active .. 0 homeless 2024-03-29T11:56:55.105 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.084+0000 7fd4f8ff9640 10 client.4559.objecter ms_handle_connect 0x7fd4e806dec0 2024-03-29T11:56:55.106 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.094+0000 7fd4f8ff9640 1 -- 172.21.15.98:0/2781976342 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd4f004fff0 con 0x7fd4fc0a9d70 2024-03-29T11:56:55.106 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.094+0000 7fd4f8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:56:55.106 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.094+0000 7fd4f8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:56:55.471 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.462+0000 7fd5028e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:56:55.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.462+0000 7fd5028e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:56:55.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.462+0000 7fd5028e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:56:55.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.462+0000 7fd5028e2640 1 -- 172.21.15.98:0/2781976342 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd4dc005330 con 0x7fd4fc0a9d70 2024-03-29T11:56:55.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.463+0000 7fd4f8ff9640 1 -- 172.21.15.98:0/2781976342 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7fd4f00520f0 con 0x7fd4fc0a9d70 2024-03-29T11:56:55.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.463+0000 7fd4f8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:56:55.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.463+0000 7fd4f8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:56:55.473 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:56:55.473 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:56:55.478 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.467+0000 7fd5028e2640 10 librados: watch_flush enter 2024-03-29T11:56:55.478 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.468+0000 7fd5028e2640 10 librados: watch_flush exit 2024-03-29T11:56:55.478 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.468+0000 7fd5028e2640 20 client.4559.objecter shutdown clearing up homeless session... 2024-03-29T11:56:55.478 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.468+0000 7fd5028e2640 10 client.4559.objecter successfully canceled tick 2024-03-29T11:56:55.478 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.468+0000 7fd5028e2640 1 -- 172.21.15.98:0/2781976342 >> v1:172.21.15.98:6816/4065186974 conn(0x7fd4e806dec0 legacy=0x7fd4e8070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:55.479 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.468+0000 7fd5028e2640 10 monclient: shutdown 2024-03-29T11:56:55.479 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.468+0000 7fd5028e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:56:55.479 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.468+0000 7fd5028e2640 1 -- 172.21.15.98:0/2781976342 >> v1:172.21.15.98:6790/0 conn(0x7fd4fc0a9d70 legacy=0x7fd4fc1db6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:56:55.479 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.469+0000 7fd5028e2640 1 -- 172.21.15.98:0/2781976342 shutdown_connections 2024-03-29T11:56:55.479 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.469+0000 7fd5028e2640 1 -- 172.21.15.98:0/2781976342 >> 172.21.15.98:0/2781976342 conn(0x7fd4fc066a00 msgr2=0x7fd4fc13c240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:56:55.479 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.469+0000 7fd5028e2640 1 -- 172.21.15.98:0/2781976342 shutdown_connections 2024-03-29T11:56:55.479 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.469+0000 7fd5028e2640 1 -- 172.21.15.98:0/2781976342 wait complete. 2024-03-29T11:56:55.479 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:56:55.469+0000 7fd5028e2640 1 librados: shutdown 2024-03-29T11:56:55.497 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:57:02.304 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:57:02.453 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.444+0000 7f925b7e4640 10 monclient: get_monmap_and_config 2024-03-29T11:57:02.453 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.444+0000 7f925b7e4640 10 monclient: build_initial_monmap 2024-03-29T11:57:02.453 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.444+0000 7f925b7e4640 10 monclient: monmap: 2024-03-29T11:57:02.454 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:57:02.454 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:02.454 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:57:02.446366+0000 2024-03-29T11:57:02.454 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:57:02.446366+0000 2024-03-29T11:57:02.454 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:57:02.454 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:02.454 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:57:02.454 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:57:02.454 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:57:02.454 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:02.455 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.445+0000 7f925b7e4640 1 Processor -- start 2024-03-29T11:57:02.455 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.445+0000 7f925b7e4640 1 -- start start 2024-03-29T11:57:02.456 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.445+0000 7f925b7e4640 10 monclient: init 2024-03-29T11:57:02.456 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.446+0000 7f925b7e4640 10 monclient: _reopen_session rank -1 2024-03-29T11:57:02.456 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.446+0000 7f925b7e4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:57:02.456 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.446+0000 7f925b7e4640 10 monclient(hunting): picked mon.noname-c con 0x7f925406c250 addr v1:172.21.15.121:6789/0 2024-03-29T11:57:02.456 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.446+0000 7f925b7e4640 10 monclient(hunting): picked mon.noname-b con 0x7f9254072400 addr v1:172.21.15.98:6790/0 2024-03-29T11:57:02.456 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.446+0000 7f925b7e4640 10 monclient(hunting): picked mon.noname-a con 0x7f925406c630 addr v1:172.21.15.98:6789/0 2024-03-29T11:57:02.456 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.446+0000 7f925b7e4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f925406d760 con 0x7f925406c630 2024-03-29T11:57:02.456 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.446+0000 7f925b7e4640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9254070500 con 0x7f925406c250 2024-03-29T11:57:02.456 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.446+0000 7f925b7e4640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92540aff50 con 0x7f9254072400 2024-03-29T11:57:02.456 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.446+0000 7f925b7e4640 10 monclient(hunting): _renew_subs 2024-03-29T11:57:02.456 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.446+0000 7f925b7e4640 10 monclient(hunting): authenticate will time out at 1329.532650s 2024-03-29T11:57:02.458 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.446+0000 7f9259580640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f925406c250 0x7f9254071cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:53998/0 (socket says 172.21.15.98:53998) 2024-03-29T11:57:02.458 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.446+0000 7f9259580640 1 -- 172.21.15.98:0/1511404289 learned_addr learned my addr 172.21.15.98:0/1511404289 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:57:02.458 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.447+0000 7f9253fff640 1 -- 172.21.15.98:0/1511404289 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2365417653 0 0) 0x7f925406d760 con 0x7f925406c630 2024-03-29T11:57:02.458 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.447+0000 7f9253fff640 10 monclient(hunting): _negotiate 2024-03-29T11:57:02.458 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.447+0000 7f9253fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:57:02.458 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.447+0000 7f9253fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:57:02.458 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.447+0000 7f9253fff640 10 monclient(hunting): my global_id is 4524 2024-03-29T11:57:02.458 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.447+0000 7f9253fff640 1 -- 172.21.15.98:0/1511404289 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f923c003520 con 0x7f925406c630 2024-03-29T11:57:02.458 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.447+0000 7f9253fff640 1 -- 172.21.15.98:0/1511404289 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3290231662 0 0) 0x7f92540aff50 con 0x7f9254072400 2024-03-29T11:57:02.458 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.447+0000 7f9253fff640 10 monclient(hunting): _negotiate 2024-03-29T11:57:02.458 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.447+0000 7f9253fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:57:02.458 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.447+0000 7f9253fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:57:02.459 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.447+0000 7f9253fff640 10 monclient(hunting): my global_id is 4562 2024-03-29T11:57:02.459 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.447+0000 7f9253fff640 1 -- 172.21.15.98:0/1511404289 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f925406d760 con 0x7f9254072400 2024-03-29T11:57:02.459 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.448+0000 7f9253fff640 1 -- 172.21.15.98:0/1511404289 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4083658456 0 0) 0x7f923c003520 con 0x7f925406c630 2024-03-29T11:57:02.459 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.448+0000 7f9253fff640 1 -- 172.21.15.98:0/1511404289 >> v1:172.21.15.98:6790/0 conn(0x7f9254072400 legacy=0x7f92541465b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:02.459 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.448+0000 7f9253fff640 1 -- 172.21.15.98:0/1511404289 >> v1:172.21.15.121:6789/0 conn(0x7f925406c250 legacy=0x7f9254071cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:57:02.459 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.448+0000 7f9253fff640 10 monclient: _finish_hunting 0 2024-03-29T11:57:02.459 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.448+0000 7f9253fff640 1 monclient: found mon.noname-a 2024-03-29T11:57:02.459 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.448+0000 7f9253fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:57:02.459 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.448+0000 7f9253fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T11:57:02.459 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.448+0000 7f9253fff640 1 -- 172.21.15.98:0/1511404289 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92540b00e0 con 0x7f925406c630 2024-03-29T11:57:02.459 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.448+0000 7f9253fff640 10 monclient: _finish_auth 0 2024-03-29T11:57:02.459 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.448+0000 7f9253fff640 10 monclient: _check_auth_tickets 2024-03-29T11:57:02.459 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.448+0000 7f9253fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:57:02.459 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.448+0000 7f9253fff640 1 -- 172.21.15.98:0/1511404289 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f9248001690 con 0x7f925406c630 2024-03-29T11:57:02.460 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.448+0000 7f9253fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:57:02.460 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.448+0000 7f9253fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:57:02.460 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.448+0000 7f9253fff640 10 monclient: dump: 2024-03-29T11:57:02.460 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:57:02.460 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:02.460 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:02.460 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:02.460 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:57:02.460 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:02.460 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:57:02.460 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:57:02.460 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:57:02.460 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:02.460 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.448+0000 7f925b7e4640 5 monclient: authenticate success, global_id 4524 2024-03-29T11:57:02.460 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.448+0000 7f925b7e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:57:02.461 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.449+0000 7f9253fff640 1 -- 172.21.15.98:0/1511404289 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92480017f0 con 0x7f925406c630 2024-03-29T11:57:02.461 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.449+0000 7f9253fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:57:02.462 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.449+0000 7f9253fff640 1 -- 172.21.15.98:0/1511404289 <== mon.0 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f9248001c00 con 0x7f925406c630 2024-03-29T11:57:02.462 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.449+0000 7f9253fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:57:02.462 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.449+0000 7f9253fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:57:02.462 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.449+0000 7f9253fff640 10 monclient: dump: 2024-03-29T11:57:02.462 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:57:02.462 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:02.462 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:02.462 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:02.462 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:57:02.463 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:02.463 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:57:02.463 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:57:02.463 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:57:02.463 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:02.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.449+0000 7f925b7e4640 10 monclient: get_monmap_and_config success 2024-03-29T11:57:02.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.449+0000 7f925b7e4640 4 set_mon_vals no callback set 2024-03-29T11:57:02.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.450+0000 7f925b7e4640 10 monclient: shutdown 2024-03-29T11:57:02.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.450+0000 7f925b7e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:57:02.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.450+0000 7f925b7e4640 1 -- 172.21.15.98:0/1511404289 >> v1:172.21.15.98:6789/0 conn(0x7f925406c630 legacy=0x7f9254151ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:02.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.450+0000 7f925b7e4640 1 -- 172.21.15.98:0/1511404289 shutdown_connections 2024-03-29T11:57:02.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.450+0000 7f925b7e4640 1 -- 172.21.15.98:0/1511404289 >> 172.21.15.98:0/1511404289 conn(0x7f9254067610 msgr2=0x7f92540b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:57:02.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.450+0000 7f925b7e4640 1 -- 172.21.15.98:0/1511404289 shutdown_connections 2024-03-29T11:57:02.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.450+0000 7f925b7e4640 1 -- 172.21.15.98:0/1511404289 wait complete. 2024-03-29T11:57:02.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.451+0000 7f925b7e4640 10 monclient: build_initial_monmap 2024-03-29T11:57:02.465 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.451+0000 7f925b7e4640 10 monclient: monmap: 2024-03-29T11:57:02.465 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:57:02.465 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:57:02.466 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:57:02.466 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:57:02.466 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:57:02.466 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:02.466 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:57:02.466 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:57:02.466 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:57:02.466 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:02.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.452+0000 7f925b7e4640 1 librados: starting msgr at 2024-03-29T11:57:02.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.452+0000 7f925b7e4640 1 librados: starting objecter 2024-03-29T11:57:02.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.452+0000 7f925b7e4640 1 Processor -- start 2024-03-29T11:57:02.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.452+0000 7f925b7e4640 1 -- start start 2024-03-29T11:57:02.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.452+0000 7f925b7e4640 1 librados: setting wanted keys 2024-03-29T11:57:02.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.452+0000 7f925b7e4640 1 librados: calling monclient init 2024-03-29T11:57:02.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.452+0000 7f925b7e4640 10 monclient: init 2024-03-29T11:57:02.469 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.453+0000 7f925b7e4640 10 monclient: _reopen_session rank -1 2024-03-29T11:57:02.469 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.453+0000 7f925b7e4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:57:02.469 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.453+0000 7f925b7e4640 10 monclient(hunting): picked mon.noname-c con 0x7f925406c250 addr v1:172.21.15.98:6790/0 2024-03-29T11:57:02.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.453+0000 7f925b7e4640 10 monclient(hunting): picked mon.noname-a con 0x7f925406c630 addr v1:172.21.15.98:6789/0 2024-03-29T11:57:02.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.453+0000 7f925b7e4640 10 monclient(hunting): picked mon.noname-b con 0x7f9254072400 addr v1:172.21.15.121:6789/0 2024-03-29T11:57:02.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.453+0000 7f9258d7f640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f925406c630 0x7f92540a9ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:50586/0 (socket says 172.21.15.98:50586) 2024-03-29T11:57:02.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.453+0000 7f9258d7f640 1 -- 172.21.15.98:0/721042226 learned_addr learned my addr 172.21.15.98:0/721042226 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:57:02.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.453+0000 7f925b7e4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92541dfff0 con 0x7f925406c630 2024-03-29T11:57:02.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.453+0000 7f925b7e4640 1 -- 172.21.15.98:0/721042226 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92541dfa80 con 0x7f9254072400 2024-03-29T11:57:02.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.453+0000 7f925b7e4640 1 -- 172.21.15.98:0/721042226 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92541dfc10 con 0x7f925406c250 2024-03-29T11:57:02.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.453+0000 7f925b7e4640 10 monclient(hunting): _renew_subs 2024-03-29T11:57:02.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.453+0000 7f925b7e4640 10 monclient(hunting): authenticate will time out at 1329.539892s 2024-03-29T11:57:02.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.453+0000 7f9251ffb640 10 client.?.objecter ms_handle_connect 0x7f925406c630 2024-03-29T11:57:02.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.453+0000 7f9251ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:57:02.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.453+0000 7f9251ffb640 10 client.?.objecter ms_handle_connect 0x7f9254072400 2024-03-29T11:57:02.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.453+0000 7f9251ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:57:02.471 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.454+0000 7f9251ffb640 10 client.?.objecter ms_handle_connect 0x7f925406c250 2024-03-29T11:57:02.471 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.454+0000 7f9251ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:57:02.471 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.454+0000 7f9251ffb640 1 -- 172.21.15.98:0/721042226 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1150913924 0 0) 0x7f92541dfa80 con 0x7f9254072400 2024-03-29T11:57:02.471 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.454+0000 7f9251ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:57:02.471 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.454+0000 7f9251ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:57:02.471 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.454+0000 7f9251ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:57:02.471 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.454+0000 7f9251ffb640 10 monclient(hunting): my global_id is 4528 2024-03-29T11:57:02.471 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.454+0000 7f9251ffb640 1 -- 172.21.15.98:0/721042226 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9240004eb0 con 0x7f9254072400 2024-03-29T11:57:02.471 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.454+0000 7f9251ffb640 1 -- 172.21.15.98:0/721042226 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 706928901 0 0) 0x7f92541dfc10 con 0x7f925406c250 2024-03-29T11:57:02.471 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.454+0000 7f9251ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:57:02.471 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.454+0000 7f9251ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:57:02.471 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.454+0000 7f9251ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:57:02.471 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.454+0000 7f9251ffb640 10 monclient(hunting): my global_id is 4565 2024-03-29T11:57:02.471 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.454+0000 7f9251ffb640 1 -- 172.21.15.98:0/721042226 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92541dfa80 con 0x7f925406c250 2024-03-29T11:57:02.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.454+0000 7f9251ffb640 1 -- 172.21.15.98:0/721042226 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3254026799 0 0) 0x7f9240004eb0 con 0x7f9254072400 2024-03-29T11:57:02.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.454+0000 7f9251ffb640 1 -- 172.21.15.98:0/721042226 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92541dfc10 con 0x7f9254072400 2024-03-29T11:57:02.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.454+0000 7f9251ffb640 1 -- 172.21.15.98:0/721042226 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f9248001b40 con 0x7f9254072400 2024-03-29T11:57:02.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.454+0000 7f9251ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:57:02.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.454+0000 7f9251ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:57:02.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.454+0000 7f9251ffb640 10 monclient(hunting): dump: 2024-03-29T11:57:02.472 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:57:02.472 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:02.472 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:02.472 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:02.472 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:57:02.472 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:02.472 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:57:02.472 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:57:02.472 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:57:02.473 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:02.473 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.454+0000 7f9251ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:57:02.473 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.455+0000 7f9251ffb640 1 -- 172.21.15.98:0/721042226 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2309694229 0 0) 0x7f92541dfa80 con 0x7f925406c250 2024-03-29T11:57:02.473 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.455+0000 7f9251ffb640 1 -- 172.21.15.98:0/721042226 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9240004eb0 con 0x7f925406c250 2024-03-29T11:57:02.473 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.455+0000 7f9251ffb640 1 -- 172.21.15.98:0/721042226 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f924c0014d0 con 0x7f925406c250 2024-03-29T11:57:02.473 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.455+0000 7f9251ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:57:02.473 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.455+0000 7f9251ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:57:02.473 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.455+0000 7f9251ffb640 10 monclient(hunting): dump: 2024-03-29T11:57:02.473 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:57:02.473 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:02.473 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:02.473 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:02.473 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:57:02.473 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:02.473 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:57:02.473 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:57:02.474 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:57:02.474 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:02.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.455+0000 7f9251ffb640 1 -- 172.21.15.98:0/721042226 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2084457140 0 0) 0x7f92541dfff0 con 0x7f925406c630 2024-03-29T11:57:02.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.455+0000 7f9251ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:57:02.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.455+0000 7f9251ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:57:02.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.455+0000 7f9251ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:57:02.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.455+0000 7f9251ffb640 10 monclient(hunting): my global_id is 4527 2024-03-29T11:57:02.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.455+0000 7f9251ffb640 1 -- 172.21.15.98:0/721042226 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92541dfa80 con 0x7f925406c630 2024-03-29T11:57:02.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.455+0000 7f9251ffb640 1 -- 172.21.15.98:0/721042226 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1702970540 0 0) 0x7f9240004eb0 con 0x7f925406c250 2024-03-29T11:57:02.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.455+0000 7f9251ffb640 1 -- 172.21.15.98:0/721042226 >> v1:172.21.15.121:6789/0 conn(0x7f9254072400 legacy=0x7f92540aa700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:02.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.455+0000 7f9251ffb640 1 -- 172.21.15.98:0/721042226 >> v1:172.21.15.98:6789/0 conn(0x7f925406c630 legacy=0x7f92540a9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:02.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.455+0000 7f9251ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:57:02.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.455+0000 7f9251ffb640 1 monclient: found mon.c 2024-03-29T11:57:02.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.455+0000 7f9251ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:57:02.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.455+0000 7f9251ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:57:02.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.455+0000 7f9251ffb640 1 -- 172.21.15.98:0/721042226 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92540ae970 con 0x7f925406c250 2024-03-29T11:57:02.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.455+0000 7f9251ffb640 10 monclient: _finish_auth 0 2024-03-29T11:57:02.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.455+0000 7f9251ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:57:02.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.455+0000 7f9251ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:57:02.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.455+0000 7f925b7e4640 5 monclient: authenticate success, global_id 4565 2024-03-29T11:57:02.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.455+0000 7f925b7e4640 10 monclient: _renew_subs 2024-03-29T11:57:02.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.455+0000 7f925b7e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:57:02.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.455+0000 7f925b7e4640 1 -- 172.21.15.98:0/721042226 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92540aec20 con 0x7f925406c250 2024-03-29T11:57:02.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.455+0000 7f925b7e4640 10 client.4565.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:57:02.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.455+0000 7f925b7e4640 10 monclient: _renew_subs 2024-03-29T11:57:02.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.456+0000 7f925b7e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:57:02.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.456+0000 7f925b7e4640 1 -- 172.21.15.98:0/721042226 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92541e78a0 con 0x7f925406c250 2024-03-29T11:57:02.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.456+0000 7f925b7e4640 1 librados: init done 2024-03-29T11:57:02.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.456+0000 7f9251ffb640 1 -- 172.21.15.98:0/721042226 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f924c001630 con 0x7f925406c250 2024-03-29T11:57:02.476 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.456+0000 7f9251ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:57:02.476 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.456+0000 7f9251ffb640 1 -- 172.21.15.98:0/721042226 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f924c001d20 con 0x7f925406c250 2024-03-29T11:57:02.476 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.456+0000 7f9251ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:57:02.476 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.456+0000 7f9251ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:57:02.476 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.456+0000 7f9251ffb640 10 monclient: dump: 2024-03-29T11:57:02.476 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:57:02.476 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:02.476 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:02.476 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:02.476 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:57:02.476 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:02.476 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:57:02.476 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:57:02.476 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:57:02.477 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:02.477 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.456+0000 7f9252ffd640 4 set_mon_vals no callback set 2024-03-29T11:57:02.477 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.457+0000 7f925b7e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:57:02.477 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.457+0000 7f925b7e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:57:02.477 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.457+0000 7f925b7e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:57:02.477 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.457+0000 7f925b7e4640 1 -- 172.21.15.98:0/721042226 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9234005080 con 0x7f925406c250 2024-03-29T11:57:02.477 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.458+0000 7f9251ffb640 1 -- 172.21.15.98:0/721042226 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f924c01a980 con 0x7f925406c250 2024-03-29T11:57:02.477 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.458+0000 7f9251ffb640 1 -- 172.21.15.98:0/721042226 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f924c084990 con 0x7f925406c250 2024-03-29T11:57:02.477 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.458+0000 7f9251ffb640 10 client.4565.objecter ms_dispatch 0x7f9254000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:57:02.477 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.458+0000 7f9251ffb640 3 client.4565.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:57:02.477 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.458+0000 7f9251ffb640 3 client.4565.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:57:02.477 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.458+0000 7f9251ffb640 20 client.4565.objecter dump_active .. 0 homeless 2024-03-29T11:57:02.478 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.459+0000 7f9258d7f640 10 monclient: get_auth_request con 0x7f924006ded0 auth_method 0 2024-03-29T11:57:02.478 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.460+0000 7f9251ffb640 10 client.4565.objecter ms_handle_connect 0x7f924006ded0 2024-03-29T11:57:02.478 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.466+0000 7f9251ffb640 1 -- 172.21.15.98:0/721042226 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f924c0500a0 con 0x7f925406c250 2024-03-29T11:57:02.478 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.467+0000 7f9251ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:57:02.478 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.467+0000 7f9251ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:57:02.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.847+0000 7f925b7e4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:57:02.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.847+0000 7f925b7e4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:57:02.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.847+0000 7f925b7e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:57:02.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.848+0000 7f925b7e4640 1 -- 172.21.15.98:0/721042226 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9234005330 con 0x7f925406c250 2024-03-29T11:57:02.858 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.849+0000 7f9251ffb640 1 -- 172.21.15.98:0/721042226 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f924c0521a0 con 0x7f925406c250 2024-03-29T11:57:02.858 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.849+0000 7f9251ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:57:02.858 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.849+0000 7f9251ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:57:02.858 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:57:02.858 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:57:02.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.852+0000 7f925b7e4640 10 librados: watch_flush enter 2024-03-29T11:57:02.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.852+0000 7f925b7e4640 10 librados: watch_flush exit 2024-03-29T11:57:02.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.852+0000 7f925b7e4640 20 client.4565.objecter shutdown clearing up homeless session... 2024-03-29T11:57:02.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.852+0000 7f925b7e4640 10 client.4565.objecter successfully canceled tick 2024-03-29T11:57:02.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.852+0000 7f925b7e4640 1 -- 172.21.15.98:0/721042226 >> v1:172.21.15.98:6816/4065186974 conn(0x7f924006ded0 legacy=0x7f9240070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:02.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.852+0000 7f925b7e4640 10 monclient: shutdown 2024-03-29T11:57:02.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.852+0000 7f925b7e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:57:02.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.852+0000 7f925b7e4640 1 -- 172.21.15.98:0/721042226 >> v1:172.21.15.98:6790/0 conn(0x7f925406c250 legacy=0x7f92541df370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:02.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.852+0000 7f925b7e4640 1 -- 172.21.15.98:0/721042226 shutdown_connections 2024-03-29T11:57:02.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.852+0000 7f925b7e4640 1 -- 172.21.15.98:0/721042226 >> 172.21.15.98:0/721042226 conn(0x7f9254067610 msgr2=0x7f9254070920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:57:02.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.853+0000 7f925b7e4640 1 -- 172.21.15.98:0/721042226 shutdown_connections 2024-03-29T11:57:02.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.853+0000 7f925b7e4640 1 -- 172.21.15.98:0/721042226 wait complete. 2024-03-29T11:57:02.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:02.853+0000 7f925b7e4640 1 librados: shutdown 2024-03-29T11:57:02.876 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:57:09.683 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:57:09.832 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.823+0000 7f5a49271640 10 monclient: get_monmap_and_config 2024-03-29T11:57:09.832 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.823+0000 7f5a49271640 10 monclient: build_initial_monmap 2024-03-29T11:57:09.832 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.823+0000 7f5a49271640 10 monclient: monmap: 2024-03-29T11:57:09.832 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:57:09.832 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:09.832 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:57:09.824736+0000 2024-03-29T11:57:09.832 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:57:09.824736+0000 2024-03-29T11:57:09.832 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:57:09.832 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:09.832 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:57:09.832 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:57:09.833 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:57:09.833 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:09.833 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.825+0000 7f5a49271640 1 Processor -- start 2024-03-29T11:57:09.833 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.825+0000 7f5a49271640 1 -- start start 2024-03-29T11:57:09.833 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.825+0000 7f5a49271640 10 monclient: init 2024-03-29T11:57:09.834 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.826+0000 7f5a49271640 10 monclient: _reopen_session rank -1 2024-03-29T11:57:09.834 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.826+0000 7f5a49271640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:57:09.837 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.829+0000 7f5a49271640 10 monclient(hunting): picked mon.noname-c con 0x7f5a440a9750 addr v1:172.21.15.121:6789/0 2024-03-29T11:57:09.838 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.830+0000 7f5a43fff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f5a440a9750 0x7f5a440ac3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:45275/0 (socket says 172.21.15.98:45275) 2024-03-29T11:57:09.838 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.830+0000 7f5a43fff640 1 -- 172.21.15.98:0/2697752927 learned_addr learned my addr 172.21.15.98:0/2697752927 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:57:09.838 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.830+0000 7f5a49271640 10 monclient(hunting): picked mon.noname-b con 0x7f5a440aad20 addr v1:172.21.15.98:6790/0 2024-03-29T11:57:09.839 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.831+0000 7f5a49271640 10 monclient(hunting): picked mon.noname-a con 0x7f5a440a9b30 addr v1:172.21.15.98:6789/0 2024-03-29T11:57:09.839 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.831+0000 7f5a49271640 1 -- 172.21.15.98:0/2697752927 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a440af4d0 con 0x7f5a440a9b30 2024-03-29T11:57:09.839 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.831+0000 7f5a49271640 1 -- 172.21.15.98:0/2697752927 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a440af7b0 con 0x7f5a440a9750 2024-03-29T11:57:09.839 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.831+0000 7f5a49271640 1 -- 172.21.15.98:0/2697752927 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a440acab0 con 0x7f5a440aad20 2024-03-29T11:57:09.839 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.831+0000 7f5a49271640 10 monclient(hunting): _renew_subs 2024-03-29T11:57:09.839 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.831+0000 7f5a49271640 10 monclient(hunting): authenticate will time out at 1336.916460s 2024-03-29T11:57:09.840 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.832+0000 7f5a42ffd640 1 -- 172.21.15.98:0/2697752927 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3901101299 0 0) 0x7f5a440acab0 con 0x7f5a440aad20 2024-03-29T11:57:09.840 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.832+0000 7f5a42ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:57:09.840 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.832+0000 7f5a42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:57:09.840 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.832+0000 7f5a42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:57:09.841 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.832+0000 7f5a42ffd640 10 monclient(hunting): my global_id is 4568 2024-03-29T11:57:09.841 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.832+0000 7f5a42ffd640 1 -- 172.21.15.98:0/2697752927 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a2c003520 con 0x7f5a440aad20 2024-03-29T11:57:09.841 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.832+0000 7f5a42ffd640 1 -- 172.21.15.98:0/2697752927 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2795970277 0 0) 0x7f5a440af7b0 con 0x7f5a440a9750 2024-03-29T11:57:09.841 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.832+0000 7f5a42ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:57:09.841 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.832+0000 7f5a42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:57:09.841 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.832+0000 7f5a42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:57:09.841 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.832+0000 7f5a42ffd640 10 monclient(hunting): my global_id is 4531 2024-03-29T11:57:09.841 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.832+0000 7f5a42ffd640 1 -- 172.21.15.98:0/2697752927 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a440acab0 con 0x7f5a440a9750 2024-03-29T11:57:09.843 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.833+0000 7f5a42ffd640 1 -- 172.21.15.98:0/2697752927 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2465418258 0 0) 0x7f5a2c003520 con 0x7f5a440aad20 2024-03-29T11:57:09.843 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.833+0000 7f5a42ffd640 1 -- 172.21.15.98:0/2697752927 >> v1:172.21.15.121:6789/0 conn(0x7f5a440a9750 legacy=0x7f5a440ac3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:09.843 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.833+0000 7f5a42ffd640 1 -- 172.21.15.98:0/2697752927 >> v1:172.21.15.98:6789/0 conn(0x7f5a440a9b30 legacy=0x7f5a44149930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:57:09.843 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.833+0000 7f5a42ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:57:09.843 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.833+0000 7f5a42ffd640 1 monclient: found mon.noname-b 2024-03-29T11:57:09.843 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.833+0000 7f5a42ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:57:09.843 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.833+0000 7f5a42ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:57:09.843 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.833+0000 7f5a42ffd640 1 -- 172.21.15.98:0/2697752927 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a440aba30 con 0x7f5a440aad20 2024-03-29T11:57:09.843 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.833+0000 7f5a42ffd640 10 monclient: _finish_auth 0 2024-03-29T11:57:09.843 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.834+0000 7f5a42ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:57:09.844 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.834+0000 7f5a42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:57:09.844 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.834+0000 7f5a42ffd640 1 -- 172.21.15.98:0/2697752927 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f5a380015c0 con 0x7f5a440aad20 2024-03-29T11:57:09.844 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.834+0000 7f5a42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:57:09.844 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.834+0000 7f5a42ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:57:09.844 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.834+0000 7f5a42ffd640 10 monclient: dump: 2024-03-29T11:57:09.844 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:57:09.844 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:09.844 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:09.844 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:09.844 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:57:09.844 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:09.844 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:57:09.844 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:57:09.844 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:57:09.845 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:09.845 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.834+0000 7f5a49271640 5 monclient: authenticate success, global_id 4568 2024-03-29T11:57:09.845 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.834+0000 7f5a49271640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:57:09.846 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.837+0000 7f5a42ffd640 1 -- 172.21.15.98:0/2697752927 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a380017a0 con 0x7f5a440aad20 2024-03-29T11:57:09.846 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.837+0000 7f5a42ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:57:09.846 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.838+0000 7f5a42ffd640 1 -- 172.21.15.98:0/2697752927 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f5a38001b70 con 0x7f5a440aad20 2024-03-29T11:57:09.846 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.838+0000 7f5a42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:57:09.846 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.838+0000 7f5a42ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:57:09.846 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.838+0000 7f5a42ffd640 10 monclient: dump: 2024-03-29T11:57:09.846 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:57:09.846 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:09.847 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:09.847 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:09.847 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:57:09.847 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:09.847 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:57:09.847 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:57:09.847 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:57:09.847 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:09.847 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.838+0000 7f5a49271640 10 monclient: get_monmap_and_config success 2024-03-29T11:57:09.847 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.838+0000 7f5a49271640 4 set_mon_vals no callback set 2024-03-29T11:57:09.848 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.839+0000 7f5a49271640 10 monclient: shutdown 2024-03-29T11:57:09.848 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.839+0000 7f5a49271640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:57:09.848 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.839+0000 7f5a49271640 1 -- 172.21.15.98:0/2697752927 >> v1:172.21.15.98:6790/0 conn(0x7f5a440aad20 legacy=0x7f5a44141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:09.848 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.839+0000 7f5a49271640 1 -- 172.21.15.98:0/2697752927 shutdown_connections 2024-03-29T11:57:09.848 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.839+0000 7f5a49271640 1 -- 172.21.15.98:0/2697752927 >> 172.21.15.98:0/2697752927 conn(0x7f5a440673e0 msgr2=0x7f5a440b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:57:09.849 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.841+0000 7f5a49271640 1 -- 172.21.15.98:0/2697752927 shutdown_connections 2024-03-29T11:57:09.849 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.841+0000 7f5a49271640 1 -- 172.21.15.98:0/2697752927 wait complete. 2024-03-29T11:57:09.850 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.842+0000 7f5a49271640 10 monclient: build_initial_monmap 2024-03-29T11:57:09.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.842+0000 7f5a49271640 10 monclient: monmap: 2024-03-29T11:57:09.851 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:57:09.851 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:57:09.851 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:57:09.851 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:57:09.851 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:57:09.851 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:09.851 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:57:09.851 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:57:09.851 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:57:09.851 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:09.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.843+0000 7f5a49271640 1 librados: starting msgr at 2024-03-29T11:57:09.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.843+0000 7f5a49271640 1 librados: starting objecter 2024-03-29T11:57:09.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.843+0000 7f5a49271640 1 Processor -- start 2024-03-29T11:57:09.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.844+0000 7f5a49271640 1 -- start start 2024-03-29T11:57:09.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.844+0000 7f5a49271640 1 librados: setting wanted keys 2024-03-29T11:57:09.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.844+0000 7f5a49271640 1 librados: calling monclient init 2024-03-29T11:57:09.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.844+0000 7f5a49271640 10 monclient: init 2024-03-29T11:57:09.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.844+0000 7f5a49271640 10 monclient: _reopen_session rank -1 2024-03-29T11:57:09.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.844+0000 7f5a49271640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:57:09.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.845+0000 7f5a49271640 10 monclient(hunting): picked mon.noname-a con 0x7f5a440a9750 addr v1:172.21.15.98:6789/0 2024-03-29T11:57:09.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.845+0000 7f5a49271640 10 monclient(hunting): picked mon.noname-c con 0x7f5a440a9b30 addr v1:172.21.15.98:6790/0 2024-03-29T11:57:09.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.845+0000 7f5a49271640 10 monclient(hunting): picked mon.noname-b con 0x7f5a440aad20 addr v1:172.21.15.121:6789/0 2024-03-29T11:57:09.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.845+0000 7f5a49271640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a4406e1c0 con 0x7f5a440a9750 2024-03-29T11:57:09.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.845+0000 7f5a49271640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a4413b360 con 0x7f5a440aad20 2024-03-29T11:57:09.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.845+0000 7f5a49271640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a44071930 con 0x7f5a440a9b30 2024-03-29T11:57:09.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.845+0000 7f5a49271640 10 monclient(hunting): _renew_subs 2024-03-29T11:57:09.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.845+0000 7f5a49271640 10 monclient(hunting): authenticate will time out at 1336.930578s 2024-03-29T11:57:09.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.846+0000 7f5a437fe640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f5a440a9b30 0x7f5a4413a540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:56746/0 (socket says 172.21.15.98:56746) 2024-03-29T11:57:09.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.846+0000 7f5a437fe640 1 -- 172.21.15.98:0/2308109035 learned_addr learned my addr 172.21.15.98:0/2308109035 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:57:09.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.846+0000 7f5a40ff9640 10 client.?.objecter ms_handle_connect 0x7f5a440a9b30 2024-03-29T11:57:09.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.846+0000 7f5a40ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:57:09.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.847+0000 7f5a40ff9640 1 -- 172.21.15.98:0/2308109035 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 371812336 0 0) 0x7f5a44071930 con 0x7f5a440a9b30 2024-03-29T11:57:09.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.847+0000 7f5a40ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:57:09.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.847+0000 7f5a40ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:57:09.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.847+0000 7f5a40ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:57:09.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.847+0000 7f5a40ff9640 10 monclient(hunting): my global_id is 4571 2024-03-29T11:57:09.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.847+0000 7f5a40ff9640 1 -- 172.21.15.98:0/2308109035 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a30004eb0 con 0x7f5a440a9b30 2024-03-29T11:57:09.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.848+0000 7f5a40ff9640 1 -- 172.21.15.98:0/2308109035 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1642251404 0 0) 0x7f5a30004eb0 con 0x7f5a440a9b30 2024-03-29T11:57:09.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.848+0000 7f5a40ff9640 1 -- 172.21.15.98:0/2308109035 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a44071930 con 0x7f5a440a9b30 2024-03-29T11:57:09.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.848+0000 7f5a40ff9640 1 -- 172.21.15.98:0/2308109035 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f5a38001b70 con 0x7f5a440a9b30 2024-03-29T11:57:09.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.848+0000 7f5a40ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:57:09.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.848+0000 7f5a40ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:57:09.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.848+0000 7f5a40ff9640 10 monclient(hunting): dump: 2024-03-29T11:57:09.864 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:57:09.864 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:09.864 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:09.864 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:09.864 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:57:09.864 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:09.864 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:57:09.864 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:57:09.864 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:57:09.864 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:09.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.848+0000 7f5a40ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:57:09.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.849+0000 7f5a40ff9640 1 -- 172.21.15.98:0/2308109035 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2377036217 0 0) 0x7f5a44071930 con 0x7f5a440a9b30 2024-03-29T11:57:09.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.849+0000 7f5a40ff9640 1 -- 172.21.15.98:0/2308109035 >> v1:172.21.15.121:6789/0 conn(0x7f5a440aad20 legacy=0x7f5a4413ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:09.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.849+0000 7f5a40ff9640 1 -- 172.21.15.98:0/2308109035 >> v1:172.21.15.98:6789/0 conn(0x7f5a440a9750 legacy=0x7f5a4406e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:09.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.849+0000 7f5a40ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:57:09.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.849+0000 7f5a40ff9640 1 monclient: found mon.c 2024-03-29T11:57:09.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.849+0000 7f5a40ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:57:09.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.849+0000 7f5a40ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:57:09.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.849+0000 7f5a40ff9640 1 -- 172.21.15.98:0/2308109035 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a4406dce0 con 0x7f5a440a9b30 2024-03-29T11:57:09.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.849+0000 7f5a40ff9640 10 monclient: _finish_auth 0 2024-03-29T11:57:09.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.849+0000 7f5a40ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:57:09.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.849+0000 7f5a40ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:57:09.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.849+0000 7f5a49271640 5 monclient: authenticate success, global_id 4571 2024-03-29T11:57:09.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.849+0000 7f5a49271640 10 monclient: _renew_subs 2024-03-29T11:57:09.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.849+0000 7f5a49271640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:57:09.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.849+0000 7f5a49271640 1 -- 172.21.15.98:0/2308109035 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a4413b680 con 0x7f5a440a9b30 2024-03-29T11:57:09.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.849+0000 7f5a49271640 10 client.4571.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:57:09.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.849+0000 7f5a49271640 10 monclient: _renew_subs 2024-03-29T11:57:09.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.849+0000 7f5a40ff9640 1 -- 172.21.15.98:0/2308109035 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a380038a0 con 0x7f5a440a9b30 2024-03-29T11:57:09.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.849+0000 7f5a49271640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:57:09.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.849+0000 7f5a49271640 1 -- 172.21.15.98:0/2308109035 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a4413ba20 con 0x7f5a440a9b30 2024-03-29T11:57:09.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.849+0000 7f5a49271640 1 librados: init done 2024-03-29T11:57:09.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.849+0000 7f5a40ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:57:09.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.849+0000 7f5a40ff9640 1 -- 172.21.15.98:0/2308109035 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f5a38003cf0 con 0x7f5a440a9b30 2024-03-29T11:57:09.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.849+0000 7f5a40ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:57:09.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.849+0000 7f5a40ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:57:09.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.849+0000 7f5a40ff9640 10 monclient: dump: 2024-03-29T11:57:09.866 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:57:09.866 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:09.866 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:09.867 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:09.867 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:57:09.867 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:09.867 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:57:09.867 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:57:09.867 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:57:09.867 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:09.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.851+0000 7f5a2a7fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:57:09.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.851+0000 7f5a2a7fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:57:09.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.851+0000 7f5a2a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:57:09.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.851+0000 7f5a2a7fc640 1 -- 172.21.15.98:0/2308109035 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a24005080 con 0x7f5a440a9b30 2024-03-29T11:57:09.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.851+0000 7f5a40ff9640 1 -- 172.21.15.98:0/2308109035 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f5a38003ed0 con 0x7f5a440a9b30 2024-03-29T11:57:09.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.852+0000 7f5a427fc640 4 set_mon_vals no callback set 2024-03-29T11:57:09.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.852+0000 7f5a40ff9640 1 -- 172.21.15.98:0/2308109035 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f5a38084e20 con 0x7f5a440a9b30 2024-03-29T11:57:09.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.852+0000 7f5a40ff9640 10 client.4571.objecter ms_dispatch 0x7f5a44000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:57:09.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.852+0000 7f5a43fff640 10 monclient: get_auth_request con 0x7f5a30071fd0 auth_method 0 2024-03-29T11:57:09.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.852+0000 7f5a40ff9640 3 client.4571.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:57:09.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.852+0000 7f5a40ff9640 3 client.4571.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:57:09.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.853+0000 7f5a40ff9640 20 client.4571.objecter dump_active .. 0 homeless 2024-03-29T11:57:09.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.853+0000 7f5a40ff9640 10 client.4571.objecter ms_handle_connect 0x7f5a30071fd0 2024-03-29T11:57:09.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.860+0000 7f5a40ff9640 1 -- 172.21.15.98:0/2308109035 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5a38050700 con 0x7f5a440a9b30 2024-03-29T11:57:09.869 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.860+0000 7f5a40ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:57:09.869 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:09.860+0000 7f5a40ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:57:10.187 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:10.179+0000 7f5a2a7fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:57:10.187 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:10.179+0000 7f5a2a7fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:57:10.187 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:10.179+0000 7f5a2a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:57:10.187 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:10.179+0000 7f5a2a7fc640 1 -- 172.21.15.98:0/2308109035 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a24005330 con 0x7f5a440a9b30 2024-03-29T11:57:10.191 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:57:10.191 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:57:10.191 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:10.180+0000 7f5a40ff9640 1 -- 172.21.15.98:0/2308109035 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f5a38052630 con 0x7f5a440a9b30 2024-03-29T11:57:10.191 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:10.180+0000 7f5a40ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:57:10.191 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:10.180+0000 7f5a40ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:57:10.192 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:10.184+0000 7f5a2a7fc640 10 librados: watch_flush enter 2024-03-29T11:57:10.192 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:10.184+0000 7f5a2a7fc640 10 librados: watch_flush exit 2024-03-29T11:57:10.192 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:10.184+0000 7f5a2a7fc640 20 client.4571.objecter shutdown clearing up homeless session... 2024-03-29T11:57:10.192 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:10.184+0000 7f5a2a7fc640 10 client.4571.objecter successfully canceled tick 2024-03-29T11:57:10.193 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:10.184+0000 7f5a2a7fc640 1 -- 172.21.15.98:0/2308109035 >> v1:172.21.15.98:6816/4065186974 conn(0x7f5a30071fd0 legacy=0x7f5a30074490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:10.193 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:10.184+0000 7f5a2a7fc640 10 monclient: shutdown 2024-03-29T11:57:10.193 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:10.184+0000 7f5a2a7fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:57:10.193 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:10.184+0000 7f5a2a7fc640 1 -- 172.21.15.98:0/2308109035 >> v1:172.21.15.98:6790/0 conn(0x7f5a440a9b30 legacy=0x7f5a4413a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:10.193 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:10.184+0000 7f5a2a7fc640 1 -- 172.21.15.98:0/2308109035 shutdown_connections 2024-03-29T11:57:10.193 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:10.184+0000 7f5a2a7fc640 1 -- 172.21.15.98:0/2308109035 >> 172.21.15.98:0/2308109035 conn(0x7f5a440673e0 msgr2=0x7f5a441417f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:57:10.193 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:10.185+0000 7f5a2a7fc640 1 -- 172.21.15.98:0/2308109035 shutdown_connections 2024-03-29T11:57:10.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:10.185+0000 7f5a2a7fc640 1 -- 172.21.15.98:0/2308109035 wait complete. 2024-03-29T11:57:10.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:10.185+0000 7f5a2a7fc640 1 librados: shutdown 2024-03-29T11:57:10.210 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:57:17.017 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:57:17.164 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.155+0000 7fc97ac4a640 10 monclient: get_monmap_and_config 2024-03-29T11:57:17.164 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.155+0000 7fc97ac4a640 10 monclient: build_initial_monmap 2024-03-29T11:57:17.164 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.155+0000 7fc97ac4a640 10 monclient: monmap: 2024-03-29T11:57:17.164 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:57:17.164 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:17.164 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:57:17.156170+0000 2024-03-29T11:57:17.164 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:57:17.156170+0000 2024-03-29T11:57:17.164 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:57:17.164 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:17.165 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:57:17.165 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:57:17.165 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:57:17.165 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:17.165 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.156+0000 7fc97ac4a640 1 Processor -- start 2024-03-29T11:57:17.166 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.156+0000 7fc97ac4a640 1 -- start start 2024-03-29T11:57:17.166 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.156+0000 7fc97ac4a640 10 monclient: init 2024-03-29T11:57:17.166 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.157+0000 7fc97ac4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:57:17.166 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.157+0000 7fc97ac4a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:57:17.166 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.157+0000 7fc97ac4a640 10 monclient(hunting): picked mon.noname-a con 0x7fc9740a9930 addr v1:172.21.15.98:6789/0 2024-03-29T11:57:17.166 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.157+0000 7fc97ac4a640 10 monclient(hunting): picked mon.noname-c con 0x7fc974149dc0 addr v1:172.21.15.121:6789/0 2024-03-29T11:57:17.166 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.157+0000 7fc97ac4a640 10 monclient(hunting): picked mon.noname-b con 0x7fc97414ff40 addr v1:172.21.15.98:6790/0 2024-03-29T11:57:17.166 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.157+0000 7fc97ac4a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9740aae60 con 0x7fc9740a9930 2024-03-29T11:57:17.166 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.157+0000 7fc97ac4a640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9740b4310 con 0x7fc974149dc0 2024-03-29T11:57:17.166 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.157+0000 7fc97ac4a640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9740afdd0 con 0x7fc97414ff40 2024-03-29T11:57:17.166 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.157+0000 7fc97ac4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:57:17.167 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.157+0000 7fc97ac4a640 10 monclient(hunting): authenticate will time out at 1344.242619s 2024-03-29T11:57:17.167 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.160+0000 7fc9789e6640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc9740a9930 0x7fc974143a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:35754/0 (socket says 172.21.15.98:35754) 2024-03-29T11:57:17.168 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.160+0000 7fc9789e6640 1 -- 172.21.15.98:0/3197354347 learned_addr learned my addr 172.21.15.98:0/3197354347 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:57:17.168 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.160+0000 7fc9737fe640 1 -- 172.21.15.98:0/3197354347 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3268874522 0 0) 0x7fc9740aae60 con 0x7fc9740a9930 2024-03-29T11:57:17.168 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.160+0000 7fc9737fe640 10 monclient(hunting): _negotiate 2024-03-29T11:57:17.168 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.160+0000 7fc9737fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:57:17.168 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.160+0000 7fc9737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:57:17.169 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.160+0000 7fc9737fe640 10 monclient(hunting): my global_id is 4530 2024-03-29T11:57:17.169 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.160+0000 7fc9737fe640 1 -- 172.21.15.98:0/3197354347 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc95c003520 con 0x7fc9740a9930 2024-03-29T11:57:17.169 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.160+0000 7fc9737fe640 1 -- 172.21.15.98:0/3197354347 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4195383520 0 0) 0x7fc9740b4310 con 0x7fc974149dc0 2024-03-29T11:57:17.169 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.160+0000 7fc9737fe640 10 monclient(hunting): _negotiate 2024-03-29T11:57:17.169 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.160+0000 7fc9737fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:57:17.169 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.160+0000 7fc9737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:57:17.169 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.160+0000 7fc9737fe640 10 monclient(hunting): my global_id is 4534 2024-03-29T11:57:17.169 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.160+0000 7fc9737fe640 1 -- 172.21.15.98:0/3197354347 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9740aae60 con 0x7fc974149dc0 2024-03-29T11:57:17.169 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.160+0000 7fc9737fe640 1 -- 172.21.15.98:0/3197354347 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 474249543 0 0) 0x7fc95c003520 con 0x7fc9740a9930 2024-03-29T11:57:17.169 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.160+0000 7fc9737fe640 1 -- 172.21.15.98:0/3197354347 >> v1:172.21.15.98:6790/0 conn(0x7fc97414ff40 legacy=0x7fc974152330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:57:17.172 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.162+0000 7fc9737fe640 1 -- 172.21.15.98:0/3197354347 >> v1:172.21.15.121:6789/0 conn(0x7fc974149dc0 legacy=0x7fc97414a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:17.172 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.162+0000 7fc9737fe640 10 monclient: _finish_hunting 0 2024-03-29T11:57:17.172 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.162+0000 7fc9737fe640 1 monclient: found mon.noname-a 2024-03-29T11:57:17.172 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.162+0000 7fc9737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:57:17.172 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.162+0000 7fc9737fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T11:57:17.172 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.162+0000 7fc9737fe640 1 -- 172.21.15.98:0/3197354347 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc97414a950 con 0x7fc9740a9930 2024-03-29T11:57:17.172 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.162+0000 7fc9737fe640 10 monclient: _finish_auth 0 2024-03-29T11:57:17.172 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.162+0000 7fc9737fe640 10 monclient: _check_auth_tickets 2024-03-29T11:57:17.172 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.162+0000 7fc9737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:57:17.172 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.162+0000 7fc9737fe640 1 -- 172.21.15.98:0/3197354347 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fc964001690 con 0x7fc9740a9930 2024-03-29T11:57:17.172 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.162+0000 7fc9737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:57:17.172 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.162+0000 7fc9737fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:57:17.172 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.162+0000 7fc9737fe640 10 monclient: dump: 2024-03-29T11:57:17.173 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:57:17.173 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:17.173 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:17.173 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:17.173 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:57:17.173 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:17.173 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:57:17.173 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:57:17.173 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:57:17.173 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:17.173 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.163+0000 7fc97ac4a640 5 monclient: authenticate success, global_id 4530 2024-03-29T11:57:17.173 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.163+0000 7fc97ac4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:57:17.173 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.164+0000 7fc9737fe640 1 -- 172.21.15.98:0/3197354347 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc964001870 con 0x7fc9740a9930 2024-03-29T11:57:17.173 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.164+0000 7fc9737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:57:17.173 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.164+0000 7fc9737fe640 1 -- 172.21.15.98:0/3197354347 <== mon.0 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fc964001cc0 con 0x7fc9740a9930 2024-03-29T11:57:17.174 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.164+0000 7fc9737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:57:17.174 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.164+0000 7fc9737fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:57:17.174 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.164+0000 7fc9737fe640 10 monclient: dump: 2024-03-29T11:57:17.174 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:57:17.174 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:17.174 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:17.174 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:17.174 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:57:17.174 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:17.174 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:57:17.174 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:57:17.174 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:57:17.174 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:17.175 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.164+0000 7fc97ac4a640 10 monclient: get_monmap_and_config success 2024-03-29T11:57:17.175 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.164+0000 7fc97ac4a640 4 set_mon_vals no callback set 2024-03-29T11:57:17.177 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.165+0000 7fc97ac4a640 10 monclient: shutdown 2024-03-29T11:57:17.178 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.165+0000 7fc97ac4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:57:17.178 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.165+0000 7fc97ac4a640 1 -- 172.21.15.98:0/3197354347 >> v1:172.21.15.98:6789/0 conn(0x7fc9740a9930 legacy=0x7fc974143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:17.178 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.165+0000 7fc97ac4a640 1 -- 172.21.15.98:0/3197354347 shutdown_connections 2024-03-29T11:57:17.178 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.165+0000 7fc97ac4a640 1 -- 172.21.15.98:0/3197354347 >> 172.21.15.98:0/3197354347 conn(0x7fc9740b35e0 msgr2=0x7fc9740b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:57:17.178 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.165+0000 7fc97ac4a640 1 -- 172.21.15.98:0/3197354347 shutdown_connections 2024-03-29T11:57:17.178 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.166+0000 7fc97ac4a640 1 -- 172.21.15.98:0/3197354347 wait complete. 2024-03-29T11:57:17.178 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.166+0000 7fc97ac4a640 10 monclient: build_initial_monmap 2024-03-29T11:57:17.178 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.166+0000 7fc97ac4a640 10 monclient: monmap: 2024-03-29T11:57:17.178 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:57:17.178 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:57:17.178 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:57:17.178 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:57:17.178 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:57:17.178 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:17.179 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:57:17.179 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:57:17.179 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:57:17.179 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:17.179 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.167+0000 7fc97ac4a640 1 librados: starting msgr at 2024-03-29T11:57:17.179 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.167+0000 7fc97ac4a640 1 librados: starting objecter 2024-03-29T11:57:17.179 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.167+0000 7fc97ac4a640 1 Processor -- start 2024-03-29T11:57:17.179 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.167+0000 7fc97ac4a640 1 -- start start 2024-03-29T11:57:17.179 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.167+0000 7fc97ac4a640 1 librados: setting wanted keys 2024-03-29T11:57:17.179 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.167+0000 7fc97ac4a640 1 librados: calling monclient init 2024-03-29T11:57:17.179 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.167+0000 7fc97ac4a640 10 monclient: init 2024-03-29T11:57:17.179 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.168+0000 7fc97ac4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:57:17.179 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.168+0000 7fc97ac4a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:57:17.179 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.168+0000 7fc97ac4a640 10 monclient(hunting): picked mon.noname-c con 0x7fc9740a9930 addr v1:172.21.15.98:6790/0 2024-03-29T11:57:17.179 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.168+0000 7fc97ac4a640 10 monclient(hunting): picked mon.noname-b con 0x7fc974149dc0 addr v1:172.21.15.121:6789/0 2024-03-29T11:57:17.180 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.168+0000 7fc97ac4a640 10 monclient(hunting): picked mon.noname-a con 0x7fc97414ff40 addr v1:172.21.15.98:6789/0 2024-03-29T11:57:17.180 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.168+0000 7fc97ac4a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9741e5230 con 0x7fc97414ff40 2024-03-29T11:57:17.180 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.168+0000 7fc97ac4a640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc97411b3a0 con 0x7fc974149dc0 2024-03-29T11:57:17.180 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.168+0000 7fc97ac4a640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc97411b680 con 0x7fc9740a9930 2024-03-29T11:57:17.180 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.168+0000 7fc97ac4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:57:17.180 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.168+0000 7fc97ac4a640 10 monclient(hunting): authenticate will time out at 1344.253818s 2024-03-29T11:57:17.181 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.173+0000 7fc9791e7640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc97414ff40 0x7fc9741e7c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37404/0 (socket says 172.21.15.98:37404) 2024-03-29T11:57:17.181 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.173+0000 7fc9791e7640 1 -- 172.21.15.98:0/2582446244 learned_addr learned my addr 172.21.15.98:0/2582446244 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:57:17.181 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.173+0000 7fc9717fa640 10 client.?.objecter ms_handle_connect 0x7fc97414ff40 2024-03-29T11:57:17.181 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.173+0000 7fc9717fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:57:17.181 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.173+0000 7fc9717fa640 10 client.?.objecter ms_handle_connect 0x7fc9740a9930 2024-03-29T11:57:17.182 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.173+0000 7fc9717fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:57:17.185 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.174+0000 7fc9717fa640 1 -- 172.21.15.98:0/2582446244 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1532092636 0 0) 0x7fc97411b680 con 0x7fc9740a9930 2024-03-29T11:57:17.185 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.174+0000 7fc9717fa640 10 monclient(hunting): _negotiate 2024-03-29T11:57:17.185 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.174+0000 7fc9717fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:57:17.185 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.174+0000 7fc9717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:57:17.185 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.174+0000 7fc9717fa640 10 monclient(hunting): my global_id is 4574 2024-03-29T11:57:17.185 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.174+0000 7fc9717fa640 1 -- 172.21.15.98:0/2582446244 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc960004eb0 con 0x7fc9740a9930 2024-03-29T11:57:17.185 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.174+0000 7fc9717fa640 1 -- 172.21.15.98:0/2582446244 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3580612440 0 0) 0x7fc9741e5230 con 0x7fc97414ff40 2024-03-29T11:57:17.185 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.174+0000 7fc9717fa640 10 monclient(hunting): _negotiate 2024-03-29T11:57:17.185 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.174+0000 7fc9717fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:57:17.186 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.174+0000 7fc9717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:57:17.186 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.174+0000 7fc9717fa640 10 monclient(hunting): my global_id is 4533 2024-03-29T11:57:17.186 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.175+0000 7fc9717fa640 1 -- 172.21.15.98:0/2582446244 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc97411b680 con 0x7fc97414ff40 2024-03-29T11:57:17.186 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.175+0000 7fc9717fa640 10 client.?.objecter ms_handle_connect 0x7fc974149dc0 2024-03-29T11:57:17.186 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.175+0000 7fc9717fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:57:17.186 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.175+0000 7fc9717fa640 1 -- 172.21.15.98:0/2582446244 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2602264045 0 0) 0x7fc97411b3a0 con 0x7fc974149dc0 2024-03-29T11:57:17.186 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.175+0000 7fc9717fa640 10 monclient(hunting): _negotiate 2024-03-29T11:57:17.186 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.175+0000 7fc9717fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:57:17.186 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.175+0000 7fc9717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:57:17.186 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.175+0000 7fc9717fa640 10 monclient(hunting): my global_id is 4537 2024-03-29T11:57:17.186 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.175+0000 7fc9717fa640 1 -- 172.21.15.98:0/2582446244 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9741e5230 con 0x7fc974149dc0 2024-03-29T11:57:17.186 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.175+0000 7fc9717fa640 1 -- 172.21.15.98:0/2582446244 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3594329972 0 0) 0x7fc97411b680 con 0x7fc97414ff40 2024-03-29T11:57:17.187 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.175+0000 7fc9717fa640 1 -- 172.21.15.98:0/2582446244 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc97411b3a0 con 0x7fc97414ff40 2024-03-29T11:57:17.187 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.175+0000 7fc9717fa640 1 -- 172.21.15.98:0/2582446244 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fc96c001a90 con 0x7fc97414ff40 2024-03-29T11:57:17.187 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.175+0000 7fc9717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:57:17.187 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.175+0000 7fc9717fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:57:17.187 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.175+0000 7fc9717fa640 10 monclient(hunting): dump: 2024-03-29T11:57:17.187 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:57:17.187 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:17.187 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:17.187 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:17.187 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:57:17.187 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:17.187 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:57:17.187 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:57:17.187 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:57:17.187 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:17.188 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.175+0000 7fc9717fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:57:17.188 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.176+0000 7fc9717fa640 1 -- 172.21.15.98:0/2582446244 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 763167097 0 0) 0x7fc9741e5230 con 0x7fc974149dc0 2024-03-29T11:57:17.188 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.176+0000 7fc9717fa640 1 -- 172.21.15.98:0/2582446244 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc97411b680 con 0x7fc974149dc0 2024-03-29T11:57:17.188 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.176+0000 7fc9717fa640 1 -- 172.21.15.98:0/2582446244 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fc9680016b0 con 0x7fc974149dc0 2024-03-29T11:57:17.188 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.176+0000 7fc9717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:57:17.188 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.176+0000 7fc9717fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:57:17.188 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.176+0000 7fc9717fa640 10 monclient(hunting): dump: 2024-03-29T11:57:17.188 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:57:17.188 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:17.188 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:17.188 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:17.188 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:57:17.188 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:17.188 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:57:17.188 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:57:17.189 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:57:17.189 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:17.190 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.178+0000 7fc9717fa640 1 -- 172.21.15.98:0/2582446244 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4148103221 0 0) 0x7fc97411b680 con 0x7fc974149dc0 2024-03-29T11:57:17.190 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.178+0000 7fc9717fa640 1 -- 172.21.15.98:0/2582446244 >> v1:172.21.15.98:6790/0 conn(0x7fc9740a9930 legacy=0x7fc9741e6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:17.191 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.178+0000 7fc9717fa640 1 -- 172.21.15.98:0/2582446244 >> v1:172.21.15.98:6789/0 conn(0x7fc97414ff40 legacy=0x7fc9741e7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:17.191 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.178+0000 7fc9717fa640 10 monclient: _finish_hunting 0 2024-03-29T11:57:17.191 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.178+0000 7fc9717fa640 1 monclient: found mon.b 2024-03-29T11:57:17.191 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.178+0000 7fc9717fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:57:17.191 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.178+0000 7fc9717fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:57:17.191 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.178+0000 7fc9717fa640 1 -- 172.21.15.98:0/2582446244 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc97411b960 con 0x7fc974149dc0 2024-03-29T11:57:17.191 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.178+0000 7fc9717fa640 10 monclient: _finish_auth 0 2024-03-29T11:57:17.191 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.178+0000 7fc9717fa640 10 monclient: _check_auth_tickets 2024-03-29T11:57:17.191 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.178+0000 7fc9717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:57:17.191 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.178+0000 7fc97ac4a640 5 monclient: authenticate success, global_id 4537 2024-03-29T11:57:17.191 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.178+0000 7fc97ac4a640 10 monclient: _renew_subs 2024-03-29T11:57:17.191 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.178+0000 7fc97ac4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:57:17.191 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.178+0000 7fc97ac4a640 1 -- 172.21.15.98:0/2582446244 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc97413f060 con 0x7fc974149dc0 2024-03-29T11:57:17.192 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.178+0000 7fc97ac4a640 10 client.4537.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:57:17.192 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.178+0000 7fc97ac4a640 10 monclient: _renew_subs 2024-03-29T11:57:17.192 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.178+0000 7fc97ac4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:57:17.192 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.178+0000 7fc97ac4a640 1 -- 172.21.15.98:0/2582446244 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9740ac770 con 0x7fc974149dc0 2024-03-29T11:57:17.192 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.178+0000 7fc97ac4a640 1 librados: init done 2024-03-29T11:57:17.192 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.180+0000 7fc9717fa640 1 -- 172.21.15.98:0/2582446244 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc968001d20 con 0x7fc974149dc0 2024-03-29T11:57:17.192 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.180+0000 7fc9717fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:57:17.192 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.180+0000 7fc9717fa640 1 -- 172.21.15.98:0/2582446244 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fc968003790 con 0x7fc974149dc0 2024-03-29T11:57:17.192 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.180+0000 7fc9717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:57:17.192 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.180+0000 7fc9717fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:57:17.192 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.180+0000 7fc9717fa640 10 monclient: dump: 2024-03-29T11:57:17.192 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:57:17.192 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:17.192 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:17.193 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:17.193 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:57:17.193 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:17.193 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:57:17.193 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:57:17.193 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:57:17.193 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:17.193 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.181+0000 7fc97ac4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:57:17.193 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.181+0000 7fc97ac4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:57:17.193 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.181+0000 7fc97ac4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:57:17.193 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.181+0000 7fc97ac4a640 1 -- 172.21.15.98:0/2582446244 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc974075f10 con 0x7fc974149dc0 2024-03-29T11:57:17.193 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.181+0000 7fc9717fa640 1 -- 172.21.15.98:0/2582446244 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7fc9680197a0 con 0x7fc974149dc0 2024-03-29T11:57:17.193 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.182+0000 7fc9717fa640 1 -- 172.21.15.98:0/2582446244 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7fc9680837c0 con 0x7fc974149dc0 2024-03-29T11:57:17.193 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.182+0000 7fc9717fa640 10 client.4537.objecter ms_dispatch 0x7fc974000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:57:17.193 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.182+0000 7fc9717fa640 3 client.4537.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:57:17.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.182+0000 7fc9717fa640 3 client.4537.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:57:17.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.182+0000 7fc9717fa640 20 client.4537.objecter dump_active .. 0 homeless 2024-03-29T11:57:17.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.181+0000 7fc9727fc640 4 set_mon_vals no callback set 2024-03-29T11:57:17.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.185+0000 7fc9789e6640 10 monclient: get_auth_request con 0x7fc96006dfd0 auth_method 0 2024-03-29T11:57:17.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.185+0000 7fc9717fa640 10 client.4537.objecter ms_handle_connect 0x7fc96006dfd0 2024-03-29T11:57:17.197 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.189+0000 7fc9717fa640 1 -- 172.21.15.98:0/2582446244 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc96804eed0 con 0x7fc974149dc0 2024-03-29T11:57:17.197 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.189+0000 7fc9717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:57:17.197 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.189+0000 7fc9717fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:57:17.526 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.518+0000 7fc97ac4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:57:17.526 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.518+0000 7fc97ac4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:57:17.526 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.518+0000 7fc97ac4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:57:17.526 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.518+0000 7fc97ac4a640 1 -- 172.21.15.98:0/2582446244 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9740ac390 con 0x7fc974149dc0 2024-03-29T11:57:17.528 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.521+0000 7fc9717fa640 1 -- 172.21.15.98:0/2582446244 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7fc968050fd0 con 0x7fc974149dc0 2024-03-29T11:57:17.528 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.521+0000 7fc9717fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:57:17.529 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.521+0000 7fc9717fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:57:17.531 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:57:17.531 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:57:17.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.525+0000 7fc97ac4a640 10 librados: watch_flush enter 2024-03-29T11:57:17.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.526+0000 7fc97ac4a640 10 librados: watch_flush exit 2024-03-29T11:57:17.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.526+0000 7fc97ac4a640 20 client.4537.objecter shutdown clearing up homeless session... 2024-03-29T11:57:17.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.526+0000 7fc97ac4a640 10 client.4537.objecter successfully canceled tick 2024-03-29T11:57:17.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.526+0000 7fc97ac4a640 1 -- 172.21.15.98:0/2582446244 >> v1:172.21.15.98:6816/4065186974 conn(0x7fc96006dfd0 legacy=0x7fc960070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:17.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.526+0000 7fc97ac4a640 10 monclient: shutdown 2024-03-29T11:57:17.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.526+0000 7fc97ac4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:57:17.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.526+0000 7fc97ac4a640 1 -- 172.21.15.98:0/2582446244 >> v1:172.21.15.121:6789/0 conn(0x7fc974149dc0 legacy=0x7fc9741e7530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:17.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.526+0000 7fc97ac4a640 1 -- 172.21.15.98:0/2582446244 shutdown_connections 2024-03-29T11:57:17.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.526+0000 7fc97ac4a640 1 -- 172.21.15.98:0/2582446244 >> 172.21.15.98:0/2582446244 conn(0x7fc9740b35e0 msgr2=0x7fc97414a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:57:17.536 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.528+0000 7fc97ac4a640 1 -- 172.21.15.98:0/2582446244 shutdown_connections 2024-03-29T11:57:17.536 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.528+0000 7fc97ac4a640 1 -- 172.21.15.98:0/2582446244 wait complete. 2024-03-29T11:57:17.536 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:17.528+0000 7fc97ac4a640 1 librados: shutdown 2024-03-29T11:57:17.553 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:57:24.360 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:57:24.514 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.504+0000 7f9eaa386640 10 monclient: get_monmap_and_config 2024-03-29T11:57:24.514 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.504+0000 7f9eaa386640 10 monclient: build_initial_monmap 2024-03-29T11:57:24.514 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.504+0000 7f9eaa386640 10 monclient: monmap: 2024-03-29T11:57:24.514 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:57:24.514 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:24.514 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:57:24.505732+0000 2024-03-29T11:57:24.514 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:57:24.505732+0000 2024-03-29T11:57:24.514 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:57:24.514 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:24.514 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:57:24.514 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:57:24.515 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:57:24.515 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:24.515 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.506+0000 7f9eaa386640 1 Processor -- start 2024-03-29T11:57:24.515 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.506+0000 7f9eaa386640 1 -- start start 2024-03-29T11:57:24.515 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.506+0000 7f9eaa386640 10 monclient: init 2024-03-29T11:57:24.517 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.507+0000 7f9eaa386640 10 monclient: _reopen_session rank -1 2024-03-29T11:57:24.517 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.507+0000 7f9eaa386640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:57:24.517 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.507+0000 7f9eaa386640 10 monclient(hunting): picked mon.noname-a con 0x7f9e9c034850 addr v1:172.21.15.98:6789/0 2024-03-29T11:57:24.517 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.507+0000 7f9eaa386640 10 monclient(hunting): picked mon.noname-c con 0x7f9e9c02a640 addr v1:172.21.15.121:6789/0 2024-03-29T11:57:24.517 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.507+0000 7f9eaa386640 10 monclient(hunting): picked mon.noname-b con 0x7f9e9c0327d0 addr v1:172.21.15.98:6790/0 2024-03-29T11:57:24.517 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.507+0000 7f9eaa386640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e9c02c820 con 0x7f9e9c034850 2024-03-29T11:57:24.517 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.507+0000 7f9eaa386640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e9c02a120 con 0x7f9e9c02a640 2024-03-29T11:57:24.517 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.507+0000 7f9eaa386640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e9c0c75b0 con 0x7f9e9c0327d0 2024-03-29T11:57:24.517 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.507+0000 7f9eaa386640 10 monclient(hunting): _renew_subs 2024-03-29T11:57:24.517 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.507+0000 7f9eaa386640 10 monclient(hunting): authenticate will time out at 1351.592985s 2024-03-29T11:57:24.519 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.511+0000 7f9ea9b85640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f9e9c0327d0 0x7f9e9c0cf240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:42213/0 (socket says 172.21.15.98:42213) 2024-03-29T11:57:24.519 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.511+0000 7f9ea9b85640 1 -- 172.21.15.98:0/1266939532 learned_addr learned my addr 172.21.15.98:0/1266939532 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:57:24.520 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.512+0000 7f9e9bfff640 1 -- 172.21.15.98:0/1266939532 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1089337291 0 0) 0x7f9e9c0c75b0 con 0x7f9e9c0327d0 2024-03-29T11:57:24.520 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.512+0000 7f9e9bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:57:24.520 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.512+0000 7f9e9bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:57:24.520 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.512+0000 7f9e9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:57:24.520 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.512+0000 7f9e9bfff640 10 monclient(hunting): my global_id is 4577 2024-03-29T11:57:24.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.512+0000 7f9e9bfff640 1 -- 172.21.15.98:0/1266939532 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e90003520 con 0x7f9e9c0327d0 2024-03-29T11:57:24.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.512+0000 7f9e9bfff640 1 -- 172.21.15.98:0/1266939532 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1243589738 0 0) 0x7f9e90003520 con 0x7f9e9c0327d0 2024-03-29T11:57:24.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.512+0000 7f9e9bfff640 1 -- 172.21.15.98:0/1266939532 >> v1:172.21.15.121:6789/0 conn(0x7f9e9c02a640 legacy=0x7f9e9c0320c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:57:24.522 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.513+0000 7f9e9bfff640 1 -- 172.21.15.98:0/1266939532 >> v1:172.21.15.98:6789/0 conn(0x7f9e9c034850 legacy=0x7f9e9c0c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:24.522 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.513+0000 7f9e9bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:57:24.522 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.513+0000 7f9e9bfff640 1 monclient: found mon.noname-b 2024-03-29T11:57:24.522 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.513+0000 7f9e9bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:57:24.523 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.513+0000 7f9e9bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:57:24.523 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.513+0000 7f9e9bfff640 1 -- 172.21.15.98:0/1266939532 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e9c033110 con 0x7f9e9c0327d0 2024-03-29T11:57:24.523 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.513+0000 7f9e9bfff640 10 monclient: _finish_auth 0 2024-03-29T11:57:24.523 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.513+0000 7f9e9bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:57:24.523 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.513+0000 7f9e9bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:57:24.523 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.513+0000 7f9e9bfff640 1 -- 172.21.15.98:0/1266939532 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f9e94001690 con 0x7f9e9c0327d0 2024-03-29T11:57:24.523 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.513+0000 7f9e9bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:57:24.523 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.513+0000 7f9e9bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:57:24.523 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.513+0000 7f9e9bfff640 10 monclient: dump: 2024-03-29T11:57:24.523 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:57:24.523 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:24.523 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:24.523 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:24.524 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:57:24.524 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:24.524 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:57:24.524 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:57:24.524 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:57:24.524 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:24.525 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.517+0000 7f9eaa386640 5 monclient: authenticate success, global_id 4577 2024-03-29T11:57:24.525 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.517+0000 7f9eaa386640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:57:24.526 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.517+0000 7f9e9bfff640 1 -- 172.21.15.98:0/1266939532 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e94001860 con 0x7f9e9c0327d0 2024-03-29T11:57:24.526 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.517+0000 7f9e9bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:57:24.526 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.517+0000 7f9eaa386640 10 monclient: get_monmap_and_config success 2024-03-29T11:57:24.526 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.517+0000 7f9eaa386640 4 set_mon_vals no callback set 2024-03-29T11:57:24.526 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.517+0000 7f9e9bfff640 1 -- 172.21.15.98:0/1266939532 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f9e94001cb0 con 0x7f9e9c0327d0 2024-03-29T11:57:24.526 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.517+0000 7f9e9bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:57:24.526 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.517+0000 7f9e9bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:57:24.526 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.517+0000 7f9e9bfff640 10 monclient: dump: 2024-03-29T11:57:24.526 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:57:24.526 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:24.526 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:24.526 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:24.527 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:57:24.527 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:24.527 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:57:24.527 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:57:24.527 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:57:24.527 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:24.528 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.518+0000 7f9eaa386640 10 monclient: shutdown 2024-03-29T11:57:24.528 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.518+0000 7f9eaa386640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:57:24.528 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.518+0000 7f9eaa386640 1 -- 172.21.15.98:0/1266939532 >> v1:172.21.15.98:6790/0 conn(0x7f9e9c0327d0 legacy=0x7f9e9c0cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:24.528 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.518+0000 7f9eaa386640 1 -- 172.21.15.98:0/1266939532 shutdown_connections 2024-03-29T11:57:24.528 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.518+0000 7f9eaa386640 1 -- 172.21.15.98:0/1266939532 >> 172.21.15.98:0/1266939532 conn(0x7f9e9c0058f0 msgr2=0x7f9e9c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:57:24.528 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.518+0000 7f9eaa386640 1 -- 172.21.15.98:0/1266939532 shutdown_connections 2024-03-29T11:57:24.528 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.518+0000 7f9eaa386640 1 -- 172.21.15.98:0/1266939532 wait complete. 2024-03-29T11:57:24.528 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.519+0000 7f9eaa386640 10 monclient: build_initial_monmap 2024-03-29T11:57:24.530 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.519+0000 7f9eaa386640 10 monclient: monmap: 2024-03-29T11:57:24.531 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:57:24.531 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:57:24.531 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:57:24.531 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:57:24.531 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:57:24.531 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:24.531 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:57:24.531 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:57:24.531 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:57:24.531 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:24.532 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.520+0000 7f9eaa386640 1 librados: starting msgr at 2024-03-29T11:57:24.532 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.520+0000 7f9eaa386640 1 librados: starting objecter 2024-03-29T11:57:24.532 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.520+0000 7f9eaa386640 1 Processor -- start 2024-03-29T11:57:24.532 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.520+0000 7f9eaa386640 1 -- start start 2024-03-29T11:57:24.532 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.520+0000 7f9eaa386640 1 librados: setting wanted keys 2024-03-29T11:57:24.532 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.520+0000 7f9eaa386640 1 librados: calling monclient init 2024-03-29T11:57:24.532 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.520+0000 7f9eaa386640 10 monclient: init 2024-03-29T11:57:24.532 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.521+0000 7f9eaa386640 10 monclient: _reopen_session rank -1 2024-03-29T11:57:24.532 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.521+0000 7f9eaa386640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:57:24.532 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.521+0000 7f9eaa386640 10 monclient(hunting): picked mon.noname-a con 0x7f9e9c02a640 addr v1:172.21.15.98:6789/0 2024-03-29T11:57:24.532 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.521+0000 7f9eaa386640 10 monclient(hunting): picked mon.noname-b con 0x7f9e9c0327d0 addr v1:172.21.15.121:6789/0 2024-03-29T11:57:24.532 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.521+0000 7f9eaa386640 10 monclient(hunting): picked mon.noname-c con 0x7f9e9c034850 addr v1:172.21.15.98:6790/0 2024-03-29T11:57:24.533 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.521+0000 7f9eaa386640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e9c0c0e20 con 0x7f9e9c02a640 2024-03-29T11:57:24.533 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.521+0000 7f9eaa386640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e9c02e910 con 0x7f9e9c0327d0 2024-03-29T11:57:24.533 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.521+0000 7f9eaa386640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e9c02ebf0 con 0x7f9e9c034850 2024-03-29T11:57:24.533 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.521+0000 7f9eaa386640 10 monclient(hunting): _renew_subs 2024-03-29T11:57:24.533 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.521+0000 7f9eaa386640 10 monclient(hunting): authenticate will time out at 1351.606286s 2024-03-29T11:57:24.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.525+0000 7f9ea8b83640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f9e9c0327d0 0x7f9e9c168880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:42671/0 (socket says 172.21.15.98:42671) 2024-03-29T11:57:24.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.525+0000 7f9ea8b83640 1 -- 172.21.15.98:0/2076243175 learned_addr learned my addr 172.21.15.98:0/2076243175 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:57:24.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.526+0000 7f9e99ffb640 10 client.?.objecter ms_handle_connect 0x7f9e9c02a640 2024-03-29T11:57:24.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.526+0000 7f9e99ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:57:24.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.526+0000 7f9e99ffb640 1 -- 172.21.15.98:0/2076243175 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1306608230 0 0) 0x7f9e9c0c0e20 con 0x7f9e9c02a640 2024-03-29T11:57:24.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.526+0000 7f9e99ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:57:24.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.526+0000 7f9e99ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:57:24.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.526+0000 7f9e99ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:57:24.535 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.526+0000 7f9e99ffb640 10 monclient(hunting): my global_id is 4539 2024-03-29T11:57:24.535 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.526+0000 7f9e99ffb640 1 -- 172.21.15.98:0/2076243175 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e88004eb0 con 0x7f9e9c02a640 2024-03-29T11:57:24.537 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.526+0000 7f9e99ffb640 1 -- 172.21.15.98:0/2076243175 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1031696327 0 0) 0x7f9e88004eb0 con 0x7f9e9c02a640 2024-03-29T11:57:24.537 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.526+0000 7f9e99ffb640 1 -- 172.21.15.98:0/2076243175 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e9c0c0e20 con 0x7f9e9c02a640 2024-03-29T11:57:24.537 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.526+0000 7f9e99ffb640 1 -- 172.21.15.98:0/2076243175 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f9ea404fdd0 con 0x7f9e9c02a640 2024-03-29T11:57:24.537 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.526+0000 7f9e99ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:57:24.537 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.526+0000 7f9e99ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:57:24.537 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.526+0000 7f9e99ffb640 10 monclient(hunting): dump: 2024-03-29T11:57:24.537 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:57:24.537 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:24.538 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:24.538 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:24.538 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:57:24.538 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:24.538 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:57:24.538 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:57:24.538 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:57:24.538 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:24.538 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.527+0000 7f9e99ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:57:24.538 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.527+0000 7f9e99ffb640 1 -- 172.21.15.98:0/2076243175 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3052468480 0 0) 0x7f9e9c0c0e20 con 0x7f9e9c02a640 2024-03-29T11:57:24.538 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.527+0000 7f9e99ffb640 1 -- 172.21.15.98:0/2076243175 >> v1:172.21.15.98:6790/0 conn(0x7f9e9c034850 legacy=0x7f9e9c16bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:24.538 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.527+0000 7f9e99ffb640 1 -- 172.21.15.98:0/2076243175 >> v1:172.21.15.121:6789/0 conn(0x7f9e9c0327d0 legacy=0x7f9e9c168880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:24.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.527+0000 7f9e99ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:57:24.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.527+0000 7f9e99ffb640 1 monclient: found mon.a 2024-03-29T11:57:24.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.527+0000 7f9e99ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:57:24.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.527+0000 7f9e99ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:57:24.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.527+0000 7f9e99ffb640 1 -- 172.21.15.98:0/2076243175 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e9c02eed0 con 0x7f9e9c02a640 2024-03-29T11:57:24.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.527+0000 7f9e99ffb640 10 monclient: _finish_auth 0 2024-03-29T11:57:24.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.527+0000 7f9e99ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:57:24.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.527+0000 7f9e99ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:57:24.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.527+0000 7f9e99ffb640 1 -- 172.21.15.98:0/2076243175 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ea404f400 con 0x7f9e9c02a640 2024-03-29T11:57:24.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.527+0000 7f9e99ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:57:24.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.527+0000 7f9e99ffb640 1 -- 172.21.15.98:0/2076243175 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f9ea4075710 con 0x7f9e9c02a640 2024-03-29T11:57:24.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.527+0000 7f9e99ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:57:24.540 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.527+0000 7f9e99ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:57:24.540 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.527+0000 7f9e99ffb640 10 monclient: dump: 2024-03-29T11:57:24.540 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:57:24.540 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:24.540 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:24.540 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:24.540 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:57:24.540 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:24.540 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:57:24.540 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:57:24.540 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:57:24.540 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:24.541 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.527+0000 7f9e9affd640 4 set_mon_vals no callback set 2024-03-29T11:57:24.541 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.528+0000 7f9eaa386640 5 monclient: authenticate success, global_id 4539 2024-03-29T11:57:24.541 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.528+0000 7f9eaa386640 10 monclient: _renew_subs 2024-03-29T11:57:24.541 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.528+0000 7f9eaa386640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:57:24.541 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.528+0000 7f9eaa386640 1 -- 172.21.15.98:0/2076243175 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e9c02dd70 con 0x7f9e9c02a640 2024-03-29T11:57:24.541 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.528+0000 7f9eaa386640 10 client.4539.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:57:24.541 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.528+0000 7f9eaa386640 10 monclient: _renew_subs 2024-03-29T11:57:24.541 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.528+0000 7f9eaa386640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:57:24.541 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.528+0000 7f9eaa386640 1 -- 172.21.15.98:0/2076243175 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e9c0beb50 con 0x7f9e9c02a640 2024-03-29T11:57:24.541 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.528+0000 7f9eaa386640 1 librados: init done 2024-03-29T11:57:24.543 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.530+0000 7f9eaa386640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:57:24.543 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.530+0000 7f9eaa386640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:57:24.543 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.530+0000 7f9eaa386640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:57:24.543 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.530+0000 7f9eaa386640 1 -- 172.21.15.98:0/2076243175 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e9c0bee00 con 0x7f9e9c02a640 2024-03-29T11:57:24.543 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.531+0000 7f9e99ffb640 1 -- 172.21.15.98:0/2076243175 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f9ea4067bf0 con 0x7f9e9c02a640 2024-03-29T11:57:24.543 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.531+0000 7f9e99ffb640 1 -- 172.21.15.98:0/2076243175 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f9ea40fe360 con 0x7f9e9c02a640 2024-03-29T11:57:24.543 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.531+0000 7f9e99ffb640 10 client.4539.objecter ms_dispatch 0x7f9ea4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:57:24.543 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.531+0000 7f9e99ffb640 3 client.4539.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:57:24.543 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.531+0000 7f9e99ffb640 3 client.4539.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:57:24.543 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.532+0000 7f9e99ffb640 20 client.4539.objecter dump_active .. 0 homeless 2024-03-29T11:57:24.543 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.532+0000 7f9ea8b83640 10 monclient: get_auth_request con 0x7f9e8806dbc0 auth_method 0 2024-03-29T11:57:24.544 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.532+0000 7f9e99ffb640 10 client.4539.objecter ms_handle_connect 0x7f9e8806dbc0 2024-03-29T11:57:24.550 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.542+0000 7f9e99ffb640 1 -- 172.21.15.98:0/2076243175 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9ea40c9c40 con 0x7f9e9c02a640 2024-03-29T11:57:24.550 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.542+0000 7f9e99ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:57:24.550 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.542+0000 7f9e99ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:57:24.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.857+0000 7f9eaa386640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:57:24.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.857+0000 7f9eaa386640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:57:24.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.857+0000 7f9eaa386640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:57:24.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.857+0000 7f9eaa386640 1 -- 172.21.15.98:0/2076243175 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9e9c0c1020 con 0x7f9e9c02a640 2024-03-29T11:57:24.869 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.861+0000 7f9e99ffb640 1 -- 172.21.15.98:0/2076243175 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f9ea40cbb70 con 0x7f9e9c02a640 2024-03-29T11:57:24.869 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.861+0000 7f9e99ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:57:24.869 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.861+0000 7f9e99ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:57:24.870 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:57:24.870 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:57:24.874 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.865+0000 7f9eaa386640 10 librados: watch_flush enter 2024-03-29T11:57:24.874 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.865+0000 7f9eaa386640 10 librados: watch_flush exit 2024-03-29T11:57:24.874 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.865+0000 7f9eaa386640 20 client.4539.objecter shutdown clearing up homeless session... 2024-03-29T11:57:24.874 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.865+0000 7f9eaa386640 10 client.4539.objecter successfully canceled tick 2024-03-29T11:57:24.874 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.865+0000 7f9eaa386640 1 -- 172.21.15.98:0/2076243175 >> v1:172.21.15.98:6816/4065186974 conn(0x7f9e8806dbc0 legacy=0x7f9e88070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:24.874 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.865+0000 7f9eaa386640 10 monclient: shutdown 2024-03-29T11:57:24.874 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.865+0000 7f9eaa386640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:57:24.874 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.865+0000 7f9eaa386640 1 -- 172.21.15.98:0/2076243175 >> v1:172.21.15.98:6789/0 conn(0x7f9e9c02a640 legacy=0x7f9e9c0be440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:24.874 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.866+0000 7f9eaa386640 1 -- 172.21.15.98:0/2076243175 shutdown_connections 2024-03-29T11:57:24.874 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.866+0000 7f9eaa386640 1 -- 172.21.15.98:0/2076243175 >> 172.21.15.98:0/2076243175 conn(0x7f9e9c0058f0 msgr2=0x7f9e9c038200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:57:24.875 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.866+0000 7f9eaa386640 1 -- 172.21.15.98:0/2076243175 shutdown_connections 2024-03-29T11:57:24.875 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.866+0000 7f9eaa386640 1 -- 172.21.15.98:0/2076243175 wait complete. 2024-03-29T11:57:24.875 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:24.866+0000 7f9eaa386640 1 librados: shutdown 2024-03-29T11:57:24.892 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:57:31.700 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:57:31.848 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.840+0000 7f0e132e8640 10 monclient: get_monmap_and_config 2024-03-29T11:57:31.848 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.840+0000 7f0e132e8640 10 monclient: build_initial_monmap 2024-03-29T11:57:31.848 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.840+0000 7f0e132e8640 10 monclient: monmap: 2024-03-29T11:57:31.848 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:57:31.848 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:31.848 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:57:31.841170+0000 2024-03-29T11:57:31.849 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:57:31.841170+0000 2024-03-29T11:57:31.849 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:57:31.849 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:31.849 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:57:31.849 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:57:31.849 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:57:31.849 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:31.850 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.843+0000 7f0e132e8640 1 Processor -- start 2024-03-29T11:57:31.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.843+0000 7f0e132e8640 1 -- start start 2024-03-29T11:57:31.852 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.843+0000 7f0e132e8640 10 monclient: init 2024-03-29T11:57:31.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.844+0000 7f0e132e8640 10 monclient: _reopen_session rank -1 2024-03-29T11:57:31.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.844+0000 7f0e132e8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:57:31.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.844+0000 7f0e132e8640 10 monclient(hunting): picked mon.noname-b con 0x7f0e0c06d9d0 addr v1:172.21.15.98:6790/0 2024-03-29T11:57:31.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.844+0000 7f0e132e8640 10 monclient(hunting): picked mon.noname-a con 0x7f0e0c13ef30 addr v1:172.21.15.98:6789/0 2024-03-29T11:57:31.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.844+0000 7f0e132e8640 10 monclient(hunting): picked mon.noname-c con 0x7f0e0c0b07d0 addr v1:172.21.15.121:6789/0 2024-03-29T11:57:31.854 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.844+0000 7f0e132e8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e0c0aed10 con 0x7f0e0c13ef30 2024-03-29T11:57:31.854 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.844+0000 7f0e132e8640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e0c0b0c40 con 0x7f0e0c0b07d0 2024-03-29T11:57:31.854 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.844+0000 7f0e132e8640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e0c0ab1b0 con 0x7f0e0c06d9d0 2024-03-29T11:57:31.854 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.844+0000 7f0e132e8640 10 monclient(hunting): _renew_subs 2024-03-29T11:57:31.854 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.844+0000 7f0e132e8640 10 monclient(hunting): authenticate will time out at 1358.929955s 2024-03-29T11:57:31.854 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.845+0000 7f0e11084640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f0e0c06d9d0 0x7f0e0c13e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:56334/0 (socket says 172.21.15.98:56334) 2024-03-29T11:57:31.854 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.845+0000 7f0e11084640 1 -- 172.21.15.98:0/1838630022 learned_addr learned my addr 172.21.15.98:0/1838630022 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:57:31.855 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.846+0000 7f0e03fff640 1 -- 172.21.15.98:0/1838630022 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1480803501 0 0) 0x7f0e0c0ab1b0 con 0x7f0e0c06d9d0 2024-03-29T11:57:31.855 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.846+0000 7f0e03fff640 10 monclient(hunting): _negotiate 2024-03-29T11:57:31.855 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.846+0000 7f0e03fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:57:31.855 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.846+0000 7f0e03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:57:31.855 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.846+0000 7f0e03fff640 10 monclient(hunting): my global_id is 4580 2024-03-29T11:57:31.855 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.846+0000 7f0e03fff640 1 -- 172.21.15.98:0/1838630022 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0df8003520 con 0x7f0e0c06d9d0 2024-03-29T11:57:31.856 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.847+0000 7f0e03fff640 1 -- 172.21.15.98:0/1838630022 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1021295569 0 0) 0x7f0df8003520 con 0x7f0e0c06d9d0 2024-03-29T11:57:31.856 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.847+0000 7f0e03fff640 1 -- 172.21.15.98:0/1838630022 >> v1:172.21.15.121:6789/0 conn(0x7f0e0c0b07d0 legacy=0x7f0e0c14bd80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:57:31.856 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.847+0000 7f0e03fff640 1 -- 172.21.15.98:0/1838630022 >> v1:172.21.15.98:6789/0 conn(0x7f0e0c13ef30 legacy=0x7f0e0c0af1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:57:31.859 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.848+0000 7f0e03fff640 10 monclient: _finish_hunting 0 2024-03-29T11:57:31.859 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.848+0000 7f0e03fff640 1 monclient: found mon.noname-b 2024-03-29T11:57:31.859 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.848+0000 7f0e03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:57:31.859 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.848+0000 7f0e03fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:57:31.859 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.848+0000 7f0e03fff640 1 -- 172.21.15.98:0/1838630022 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e0c0ab490 con 0x7f0e0c06d9d0 2024-03-29T11:57:31.859 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.848+0000 7f0e03fff640 10 monclient: _finish_auth 0 2024-03-29T11:57:31.859 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.848+0000 7f0e03fff640 10 monclient: _check_auth_tickets 2024-03-29T11:57:31.859 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.848+0000 7f0e03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:57:31.860 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.848+0000 7f0e03fff640 1 -- 172.21.15.98:0/1838630022 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f0e04001a10 con 0x7f0e0c06d9d0 2024-03-29T11:57:31.860 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.848+0000 7f0e03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:57:31.860 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.848+0000 7f0e03fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:57:31.860 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.848+0000 7f0e03fff640 10 monclient: dump: 2024-03-29T11:57:31.860 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:57:31.860 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:31.860 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:31.860 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:31.860 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:57:31.860 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:31.860 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:57:31.860 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:57:31.860 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:57:31.860 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:31.860 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.848+0000 7f0e132e8640 5 monclient: authenticate success, global_id 4580 2024-03-29T11:57:31.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.848+0000 7f0e132e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:57:31.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.849+0000 7f0e03fff640 1 -- 172.21.15.98:0/1838630022 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e04001be0 con 0x7f0e0c06d9d0 2024-03-29T11:57:31.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.849+0000 7f0e03fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:57:31.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.849+0000 7f0e03fff640 1 -- 172.21.15.98:0/1838630022 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f0e04002030 con 0x7f0e0c06d9d0 2024-03-29T11:57:31.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.849+0000 7f0e03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:57:31.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.849+0000 7f0e03fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:57:31.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.849+0000 7f0e03fff640 10 monclient: dump: 2024-03-29T11:57:31.861 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:57:31.861 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:31.861 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:31.861 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:31.861 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:57:31.861 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:31.861 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:57:31.862 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:57:31.862 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:57:31.862 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:31.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.849+0000 7f0e132e8640 10 monclient: get_monmap_and_config success 2024-03-29T11:57:31.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.849+0000 7f0e132e8640 4 set_mon_vals no callback set 2024-03-29T11:57:31.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.850+0000 7f0e132e8640 10 monclient: shutdown 2024-03-29T11:57:31.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.850+0000 7f0e132e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:57:31.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.850+0000 7f0e132e8640 1 -- 172.21.15.98:0/1838630022 >> v1:172.21.15.98:6790/0 conn(0x7f0e0c06d9d0 legacy=0x7f0e0c13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:31.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.851+0000 7f0e132e8640 1 -- 172.21.15.98:0/1838630022 shutdown_connections 2024-03-29T11:57:31.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.851+0000 7f0e132e8640 1 -- 172.21.15.98:0/1838630022 >> 172.21.15.98:0/1838630022 conn(0x7f0e0c06b660 msgr2=0x7f0e0c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:57:31.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.852+0000 7f0e132e8640 1 -- 172.21.15.98:0/1838630022 shutdown_connections 2024-03-29T11:57:31.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.852+0000 7f0e132e8640 1 -- 172.21.15.98:0/1838630022 wait complete. 2024-03-29T11:57:31.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.854+0000 7f0e132e8640 10 monclient: build_initial_monmap 2024-03-29T11:57:31.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.854+0000 7f0e132e8640 10 monclient: monmap: 2024-03-29T11:57:31.865 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:57:31.866 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:57:31.866 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:57:31.866 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:57:31.866 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:57:31.866 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:31.866 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:57:31.866 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:57:31.866 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:57:31.866 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:31.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.854+0000 7f0e132e8640 1 librados: starting msgr at 2024-03-29T11:57:31.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.854+0000 7f0e132e8640 1 librados: starting objecter 2024-03-29T11:57:31.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.855+0000 7f0e132e8640 1 Processor -- start 2024-03-29T11:57:31.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.855+0000 7f0e132e8640 1 -- start start 2024-03-29T11:57:31.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.855+0000 7f0e132e8640 1 librados: setting wanted keys 2024-03-29T11:57:31.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.855+0000 7f0e132e8640 1 librados: calling monclient init 2024-03-29T11:57:31.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.855+0000 7f0e132e8640 10 monclient: init 2024-03-29T11:57:31.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.856+0000 7f0e132e8640 10 monclient: _reopen_session rank -1 2024-03-29T11:57:31.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.856+0000 7f0e132e8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:57:31.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.856+0000 7f0e132e8640 10 monclient(hunting): picked mon.noname-b con 0x7f0e0c06d9d0 addr v1:172.21.15.121:6789/0 2024-03-29T11:57:31.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.856+0000 7f0e132e8640 10 monclient(hunting): picked mon.noname-c con 0x7f0e0c0b07d0 addr v1:172.21.15.98:6790/0 2024-03-29T11:57:31.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.856+0000 7f0e132e8640 10 monclient(hunting): picked mon.noname-a con 0x7f0e0c13ef30 addr v1:172.21.15.98:6789/0 2024-03-29T11:57:31.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.856+0000 7f0e132e8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e0c0b38b0 con 0x7f0e0c13ef30 2024-03-29T11:57:31.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.856+0000 7f0e132e8640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e0c0b3b90 con 0x7f0e0c06d9d0 2024-03-29T11:57:31.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.856+0000 7f0e132e8640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e0c0b26e0 con 0x7f0e0c0b07d0 2024-03-29T11:57:31.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.856+0000 7f0e132e8640 10 monclient(hunting): _renew_subs 2024-03-29T11:57:31.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.856+0000 7f0e132e8640 10 monclient(hunting): authenticate will time out at 1358.941552s 2024-03-29T11:57:31.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.856+0000 7f0e10883640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f0e0c0b07d0 0x7f0e0c139fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:57728/0 (socket says 172.21.15.98:57728) 2024-03-29T11:57:31.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.856+0000 7f0e10883640 1 -- 172.21.15.98:0/2960266365 learned_addr learned my addr 172.21.15.98:0/2960266365 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:57:31.869 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.859+0000 7f0e01ffb640 10 client.?.objecter ms_handle_connect 0x7f0e0c0b07d0 2024-03-29T11:57:31.869 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.859+0000 7f0e01ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:57:31.869 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.859+0000 7f0e01ffb640 1 -- 172.21.15.98:0/2960266365 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3183046252 0 0) 0x7f0e0c0b26e0 con 0x7f0e0c0b07d0 2024-03-29T11:57:31.869 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.859+0000 7f0e01ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:57:31.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.859+0000 7f0e01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:57:31.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.859+0000 7f0e01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:57:31.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.859+0000 7f0e01ffb640 10 monclient(hunting): my global_id is 4583 2024-03-29T11:57:31.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.859+0000 7f0e01ffb640 1 -- 172.21.15.98:0/2960266365 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0df0004eb0 con 0x7f0e0c0b07d0 2024-03-29T11:57:31.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.859+0000 7f0e01ffb640 10 client.?.objecter ms_handle_connect 0x7f0e0c06d9d0 2024-03-29T11:57:31.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.859+0000 7f0e01ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:57:31.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.859+0000 7f0e01ffb640 1 -- 172.21.15.98:0/2960266365 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2391760749 0 0) 0x7f0e0c0b3b90 con 0x7f0e0c06d9d0 2024-03-29T11:57:31.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.859+0000 7f0e01ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:57:31.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.859+0000 7f0e01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:57:31.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.859+0000 7f0e01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:57:31.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.859+0000 7f0e01ffb640 10 monclient(hunting): my global_id is 4540 2024-03-29T11:57:31.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.859+0000 7f0e01ffb640 1 -- 172.21.15.98:0/2960266365 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e0c0b26e0 con 0x7f0e0c06d9d0 2024-03-29T11:57:31.871 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.860+0000 7f0e01ffb640 1 -- 172.21.15.98:0/2960266365 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2846845882 0 0) 0x7f0df0004eb0 con 0x7f0e0c0b07d0 2024-03-29T11:57:31.871 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.860+0000 7f0e01ffb640 1 -- 172.21.15.98:0/2960266365 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e0c0b3b90 con 0x7f0e0c0b07d0 2024-03-29T11:57:31.871 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.860+0000 7f0e01ffb640 1 -- 172.21.15.98:0/2960266365 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f0e08001960 con 0x7f0e0c0b07d0 2024-03-29T11:57:31.871 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.860+0000 7f0e01ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:57:31.871 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.860+0000 7f0e01ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:57:31.871 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.860+0000 7f0e01ffb640 10 monclient(hunting): dump: 2024-03-29T11:57:31.871 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:57:31.871 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:31.871 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:31.871 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:31.871 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:57:31.872 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:31.872 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:57:31.872 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:57:31.872 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:57:31.872 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:31.872 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.860+0000 7f0e01ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:57:31.872 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.860+0000 7f0e01ffb640 1 -- 172.21.15.98:0/2960266365 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 440212819 0 0) 0x7f0e0c0b3b90 con 0x7f0e0c0b07d0 2024-03-29T11:57:31.872 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.860+0000 7f0e01ffb640 1 -- 172.21.15.98:0/2960266365 >> v1:172.21.15.121:6789/0 conn(0x7f0e0c06d9d0 legacy=0x7f0e0c14dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:31.872 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.860+0000 7f0e01ffb640 1 -- 172.21.15.98:0/2960266365 >> v1:172.21.15.98:6789/0 conn(0x7f0e0c13ef30 legacy=0x7f0e0c13a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:31.872 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.860+0000 7f0e01ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:57:31.872 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.860+0000 7f0e01ffb640 1 monclient: found mon.c 2024-03-29T11:57:31.872 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.860+0000 7f0e01ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:57:31.872 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.860+0000 7f0e01ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:57:31.872 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.860+0000 7f0e01ffb640 1 -- 172.21.15.98:0/2960266365 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e0c0b3fb0 con 0x7f0e0c0b07d0 2024-03-29T11:57:31.872 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.860+0000 7f0e01ffb640 10 monclient: _finish_auth 0 2024-03-29T11:57:31.872 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.860+0000 7f0e01ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:57:31.873 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.860+0000 7f0e01ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:57:31.873 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.860+0000 7f0e132e8640 5 monclient: authenticate success, global_id 4583 2024-03-29T11:57:31.873 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.860+0000 7f0e132e8640 10 monclient: _renew_subs 2024-03-29T11:57:31.873 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.860+0000 7f0e132e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:57:31.873 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.860+0000 7f0e132e8640 1 -- 172.21.15.98:0/2960266365 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e0c0b4260 con 0x7f0e0c0b07d0 2024-03-29T11:57:31.873 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.861+0000 7f0e01ffb640 1 -- 172.21.15.98:0/2960266365 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e08001ac0 con 0x7f0e0c0b07d0 2024-03-29T11:57:31.873 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.861+0000 7f0e01ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:57:31.873 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.861+0000 7f0e037fe640 4 set_mon_vals no callback set 2024-03-29T11:57:31.873 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.861+0000 7f0e132e8640 10 client.4583.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:57:31.873 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.861+0000 7f0e132e8640 10 monclient: _renew_subs 2024-03-29T11:57:31.873 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.861+0000 7f0e132e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:57:31.873 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.861+0000 7f0e132e8640 1 -- 172.21.15.98:0/2960266365 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e0c0b1980 con 0x7f0e0c0b07d0 2024-03-29T11:57:31.873 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.861+0000 7f0e132e8640 1 librados: init done 2024-03-29T11:57:31.873 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.861+0000 7f0e01ffb640 1 -- 172.21.15.98:0/2960266365 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f0e080038a0 con 0x7f0e0c0b07d0 2024-03-29T11:57:31.873 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.861+0000 7f0e01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:57:31.874 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.861+0000 7f0e01ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:57:31.874 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.861+0000 7f0e01ffb640 10 monclient: dump: 2024-03-29T11:57:31.874 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:57:31.874 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:31.874 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:31.874 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:31.874 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:57:31.874 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:31.874 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:57:31.874 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:57:31.874 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:57:31.874 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:31.875 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.865+0000 7f0e01ffb640 1 -- 172.21.15.98:0/2960266365 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f0e08003a00 con 0x7f0e0c0b07d0 2024-03-29T11:57:31.875 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.865+0000 7f0e132e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:57:31.875 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.865+0000 7f0e132e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:57:31.875 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.865+0000 7f0e132e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:57:31.875 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.865+0000 7f0e132e8640 1 -- 172.21.15.98:0/2960266365 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e0c075f10 con 0x7f0e0c0b07d0 2024-03-29T11:57:31.875 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.866+0000 7f0e01ffb640 1 -- 172.21.15.98:0/2960266365 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f0e080849d0 con 0x7f0e0c0b07d0 2024-03-29T11:57:31.876 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.866+0000 7f0e01ffb640 10 client.4583.objecter ms_dispatch 0x7f0e0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:57:31.876 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.866+0000 7f0e01ffb640 3 client.4583.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:57:31.876 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.866+0000 7f0e01ffb640 3 client.4583.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:57:31.876 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.866+0000 7f0e01ffb640 20 client.4583.objecter dump_active .. 0 homeless 2024-03-29T11:57:31.876 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.866+0000 7f0e11084640 10 monclient: get_auth_request con 0x7f0df006dd40 auth_method 0 2024-03-29T11:57:31.876 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.866+0000 7f0e01ffb640 10 client.4583.objecter ms_handle_connect 0x7f0df006dd40 2024-03-29T11:57:31.883 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.874+0000 7f0e01ffb640 1 -- 172.21.15.98:0/2960266365 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0e080502b0 con 0x7f0e0c0b07d0 2024-03-29T11:57:31.883 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.874+0000 7f0e01ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:57:31.883 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:31.874+0000 7f0e01ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:57:32.201 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:32.193+0000 7f0e132e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:57:32.202 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:32.193+0000 7f0e132e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:57:32.202 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:32.193+0000 7f0e132e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:57:32.202 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:32.193+0000 7f0e132e8640 1 -- 172.21.15.98:0/2960266365 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e0c072330 con 0x7f0e0c0b07d0 2024-03-29T11:57:32.205 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:32.198+0000 7f0e01ffb640 1 -- 172.21.15.98:0/2960266365 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f0e080521e0 con 0x7f0e0c0b07d0 2024-03-29T11:57:32.206 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:32.198+0000 7f0e01ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:57:32.206 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:32.198+0000 7f0e01ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:57:32.206 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:57:32.207 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:57:32.210 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:32.202+0000 7f0e132e8640 10 librados: watch_flush enter 2024-03-29T11:57:32.210 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:32.202+0000 7f0e132e8640 10 librados: watch_flush exit 2024-03-29T11:57:32.210 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:32.202+0000 7f0e132e8640 20 client.4583.objecter shutdown clearing up homeless session... 2024-03-29T11:57:32.210 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:32.202+0000 7f0e132e8640 10 client.4583.objecter successfully canceled tick 2024-03-29T11:57:32.210 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:32.203+0000 7f0e132e8640 1 -- 172.21.15.98:0/2960266365 >> v1:172.21.15.98:6816/4065186974 conn(0x7f0df006dd40 legacy=0x7f0df0070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:32.210 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:32.203+0000 7f0e132e8640 10 monclient: shutdown 2024-03-29T11:57:32.211 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:32.203+0000 7f0e132e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:57:32.211 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:32.203+0000 7f0e132e8640 1 -- 172.21.15.98:0/2960266365 >> v1:172.21.15.98:6790/0 conn(0x7f0e0c0b07d0 legacy=0x7f0e0c139fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:32.211 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:32.203+0000 7f0e132e8640 1 -- 172.21.15.98:0/2960266365 shutdown_connections 2024-03-29T11:57:32.211 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:32.203+0000 7f0e132e8640 1 -- 172.21.15.98:0/2960266365 >> 172.21.15.98:0/2960266365 conn(0x7f0e0c06b660 msgr2=0x7f0e0c13e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:57:32.215 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:32.207+0000 7f0e132e8640 1 -- 172.21.15.98:0/2960266365 shutdown_connections 2024-03-29T11:57:32.215 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:32.207+0000 7f0e132e8640 1 -- 172.21.15.98:0/2960266365 wait complete. 2024-03-29T11:57:32.218 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:32.211+0000 7f0e132e8640 1 librados: shutdown 2024-03-29T11:57:32.237 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:57:39.043 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:57:39.191 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.183+0000 7fa5f3674640 10 monclient: get_monmap_and_config 2024-03-29T11:57:39.191 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.183+0000 7fa5f3674640 10 monclient: build_initial_monmap 2024-03-29T11:57:39.191 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.183+0000 7fa5f3674640 10 monclient: monmap: 2024-03-29T11:57:39.191 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:57:39.191 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:39.191 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:57:39.184203+0000 2024-03-29T11:57:39.191 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:57:39.184203+0000 2024-03-29T11:57:39.191 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:57:39.191 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:39.191 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:57:39.191 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:57:39.191 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:57:39.192 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:39.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.184+0000 7fa5f3674640 1 Processor -- start 2024-03-29T11:57:39.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.184+0000 7fa5f3674640 1 -- start start 2024-03-29T11:57:39.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.184+0000 7fa5f3674640 10 monclient: init 2024-03-29T11:57:39.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.185+0000 7fa5f3674640 10 monclient: _reopen_session rank -1 2024-03-29T11:57:39.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.185+0000 7fa5f3674640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:57:39.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.185+0000 7fa5f3674640 10 monclient(hunting): picked mon.noname-b con 0x7fa5ec1422b0 addr v1:172.21.15.98:6790/0 2024-03-29T11:57:39.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.185+0000 7fa5f3674640 10 monclient(hunting): picked mon.noname-c con 0x7fa5ec142690 addr v1:172.21.15.121:6789/0 2024-03-29T11:57:39.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.185+0000 7fa5f3674640 10 monclient(hunting): picked mon.noname-a con 0x7fa5ec0b17d0 addr v1:172.21.15.98:6789/0 2024-03-29T11:57:39.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.185+0000 7fa5f3674640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ec142c80 con 0x7fa5ec0b17d0 2024-03-29T11:57:39.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.185+0000 7fa5f3674640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ec0aae40 con 0x7fa5ec142690 2024-03-29T11:57:39.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.185+0000 7fa5f3674640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ec13fbd0 con 0x7fa5ec1422b0 2024-03-29T11:57:39.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.185+0000 7fa5f3674640 10 monclient(hunting): _renew_subs 2024-03-29T11:57:39.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.185+0000 7fa5f3674640 10 monclient(hunting): authenticate will time out at 1366.270384s 2024-03-29T11:57:39.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.185+0000 7fa5f0c0f640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fa5ec142690 0x7fa5ec0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:37698/0 (socket says 172.21.15.98:37698) 2024-03-29T11:57:39.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.185+0000 7fa5f0c0f640 1 -- 172.21.15.98:0/266907264 learned_addr learned my addr 172.21.15.98:0/266907264 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:57:39.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.186+0000 7fa5e3fff640 1 -- 172.21.15.98:0/266907264 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2304828547 0 0) 0x7fa5ec13fbd0 con 0x7fa5ec1422b0 2024-03-29T11:57:39.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.186+0000 7fa5e3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:57:39.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.186+0000 7fa5e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:57:39.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.186+0000 7fa5e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:57:39.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.186+0000 7fa5e3fff640 10 monclient(hunting): my global_id is 4586 2024-03-29T11:57:39.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.186+0000 7fa5e3fff640 1 -- 172.21.15.98:0/266907264 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5d8003520 con 0x7fa5ec1422b0 2024-03-29T11:57:39.197 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.186+0000 7fa5e3fff640 1 -- 172.21.15.98:0/266907264 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2991235118 0 0) 0x7fa5ec142c80 con 0x7fa5ec0b17d0 2024-03-29T11:57:39.197 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.186+0000 7fa5e3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:57:39.197 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.186+0000 7fa5e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:57:39.197 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.186+0000 7fa5e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:57:39.197 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.186+0000 7fa5e3fff640 10 monclient(hunting): my global_id is 4542 2024-03-29T11:57:39.197 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.186+0000 7fa5e3fff640 1 -- 172.21.15.98:0/266907264 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5ec13fbd0 con 0x7fa5ec0b17d0 2024-03-29T11:57:39.197 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.186+0000 7fa5e3fff640 1 -- 172.21.15.98:0/266907264 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 986727577 0 0) 0x7fa5d8003520 con 0x7fa5ec1422b0 2024-03-29T11:57:39.197 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.186+0000 7fa5e3fff640 1 -- 172.21.15.98:0/266907264 >> v1:172.21.15.121:6789/0 conn(0x7fa5ec142690 legacy=0x7fa5ec0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:39.197 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.186+0000 7fa5e3fff640 1 -- 172.21.15.98:0/266907264 >> v1:172.21.15.98:6789/0 conn(0x7fa5ec0b17d0 legacy=0x7fa5ec0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:39.197 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.186+0000 7fa5e3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:57:39.197 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.186+0000 7fa5e3fff640 1 monclient: found mon.noname-b 2024-03-29T11:57:39.197 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.186+0000 7fa5e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:57:39.198 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.186+0000 7fa5e3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:57:39.198 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.186+0000 7fa5e3fff640 1 -- 172.21.15.98:0/266907264 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ec151940 con 0x7fa5ec1422b0 2024-03-29T11:57:39.198 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.186+0000 7fa5e3fff640 10 monclient: _finish_auth 0 2024-03-29T11:57:39.198 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.186+0000 7fa5e3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:57:39.198 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.186+0000 7fa5e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:57:39.198 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.186+0000 7fa5e3fff640 1 -- 172.21.15.98:0/266907264 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fa5e4001a10 con 0x7fa5ec1422b0 2024-03-29T11:57:39.198 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.186+0000 7fa5e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:57:39.198 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.186+0000 7fa5e3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:57:39.198 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.186+0000 7fa5e3fff640 10 monclient: dump: 2024-03-29T11:57:39.198 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:57:39.199 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:39.199 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:39.199 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:39.199 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:57:39.199 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:39.199 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:57:39.199 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:57:39.199 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:57:39.199 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:39.199 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.186+0000 7fa5f3674640 5 monclient: authenticate success, global_id 4586 2024-03-29T11:57:39.200 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.186+0000 7fa5f3674640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:57:39.200 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.187+0000 7fa5e3fff640 1 -- 172.21.15.98:0/266907264 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5e4001b70 con 0x7fa5ec1422b0 2024-03-29T11:57:39.200 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.187+0000 7fa5e3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:57:39.200 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.187+0000 7fa5e3fff640 1 -- 172.21.15.98:0/266907264 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fa5e4001f00 con 0x7fa5ec1422b0 2024-03-29T11:57:39.200 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.187+0000 7fa5e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:57:39.200 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.187+0000 7fa5e3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:57:39.200 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.187+0000 7fa5e3fff640 10 monclient: dump: 2024-03-29T11:57:39.200 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:57:39.200 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:39.200 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:39.201 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:39.201 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:57:39.201 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:39.201 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:57:39.201 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:57:39.201 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:57:39.201 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:39.201 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.187+0000 7fa5f3674640 10 monclient: get_monmap_and_config success 2024-03-29T11:57:39.201 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.187+0000 7fa5f3674640 4 set_mon_vals no callback set 2024-03-29T11:57:39.207 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.188+0000 7fa5f3674640 10 monclient: shutdown 2024-03-29T11:57:39.207 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.188+0000 7fa5f3674640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:57:39.207 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.188+0000 7fa5f3674640 1 -- 172.21.15.98:0/266907264 >> v1:172.21.15.98:6790/0 conn(0x7fa5ec1422b0 legacy=0x7fa5ec13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:39.207 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.188+0000 7fa5f3674640 1 -- 172.21.15.98:0/266907264 shutdown_connections 2024-03-29T11:57:39.207 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.188+0000 7fa5f3674640 1 -- 172.21.15.98:0/266907264 >> 172.21.15.98:0/266907264 conn(0x7fa5ec066a00 msgr2=0x7fa5ec066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:57:39.207 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.189+0000 7fa5f3674640 1 -- 172.21.15.98:0/266907264 shutdown_connections 2024-03-29T11:57:39.208 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.189+0000 7fa5f3674640 1 -- 172.21.15.98:0/266907264 wait complete. 2024-03-29T11:57:39.208 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.189+0000 7fa5f3674640 10 monclient: build_initial_monmap 2024-03-29T11:57:39.208 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.189+0000 7fa5f3674640 10 monclient: monmap: 2024-03-29T11:57:39.208 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:57:39.208 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:57:39.208 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:57:39.208 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:57:39.208 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:57:39.208 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:39.208 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:57:39.208 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:57:39.208 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:57:39.208 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:39.208 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.190+0000 7fa5f3674640 1 librados: starting msgr at 2024-03-29T11:57:39.208 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.190+0000 7fa5f3674640 1 librados: starting objecter 2024-03-29T11:57:39.209 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.190+0000 7fa5f3674640 1 Processor -- start 2024-03-29T11:57:39.209 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.190+0000 7fa5f3674640 1 -- start start 2024-03-29T11:57:39.209 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.190+0000 7fa5f3674640 1 librados: setting wanted keys 2024-03-29T11:57:39.209 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.190+0000 7fa5f3674640 1 librados: calling monclient init 2024-03-29T11:57:39.209 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.190+0000 7fa5f3674640 10 monclient: init 2024-03-29T11:57:39.209 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.191+0000 7fa5f3674640 10 monclient: _reopen_session rank -1 2024-03-29T11:57:39.209 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.191+0000 7fa5f3674640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:57:39.209 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.191+0000 7fa5f3674640 10 monclient(hunting): picked mon.noname-b con 0x7fa5ec0b17d0 addr v1:172.21.15.121:6789/0 2024-03-29T11:57:39.209 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.191+0000 7fa5f3674640 10 monclient(hunting): picked mon.noname-a con 0x7fa5ec1422b0 addr v1:172.21.15.98:6789/0 2024-03-29T11:57:39.209 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.191+0000 7fa5f3674640 10 monclient(hunting): picked mon.noname-c con 0x7fa5ec142690 addr v1:172.21.15.98:6790/0 2024-03-29T11:57:39.209 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.191+0000 7fa5f3674640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ec070770 con 0x7fa5ec1422b0 2024-03-29T11:57:39.209 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.191+0000 7fa5f3674640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ec070a50 con 0x7fa5ec0b17d0 2024-03-29T11:57:39.209 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.191+0000 7fa5f3674640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ec070d30 con 0x7fa5ec142690 2024-03-29T11:57:39.209 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.192+0000 7fa5f3674640 10 monclient(hunting): _renew_subs 2024-03-29T11:57:39.210 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.192+0000 7fa5f3674640 10 monclient(hunting): authenticate will time out at 1366.277064s 2024-03-29T11:57:39.210 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.192+0000 7fa5f0c0f640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa5ec1422b0 0x7fa5ec06ebf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:39954/0 (socket says 172.21.15.98:39954) 2024-03-29T11:57:39.210 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.192+0000 7fa5f0c0f640 1 -- 172.21.15.98:0/3601937517 learned_addr learned my addr 172.21.15.98:0/3601937517 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:57:39.210 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.192+0000 7fa5e1ffb640 10 client.?.objecter ms_handle_connect 0x7fa5ec1422b0 2024-03-29T11:57:39.210 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.192+0000 7fa5e1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:57:39.210 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.192+0000 7fa5e1ffb640 10 client.?.objecter ms_handle_connect 0x7fa5ec142690 2024-03-29T11:57:39.210 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.192+0000 7fa5e1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:57:39.210 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.192+0000 7fa5e1ffb640 1 -- 172.21.15.98:0/3601937517 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3683050034 0 0) 0x7fa5ec070d30 con 0x7fa5ec142690 2024-03-29T11:57:39.210 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.192+0000 7fa5e1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:57:39.210 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.192+0000 7fa5e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:57:39.210 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.192+0000 7fa5e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:57:39.210 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.192+0000 7fa5e1ffb640 10 monclient(hunting): my global_id is 4589 2024-03-29T11:57:39.210 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.192+0000 7fa5e1ffb640 1 -- 172.21.15.98:0/3601937517 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5d0004eb0 con 0x7fa5ec142690 2024-03-29T11:57:39.210 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.193+0000 7fa5e1ffb640 1 -- 172.21.15.98:0/3601937517 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 793486767 0 0) 0x7fa5ec070770 con 0x7fa5ec1422b0 2024-03-29T11:57:39.211 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.193+0000 7fa5e1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:57:39.211 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.193+0000 7fa5e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:57:39.211 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.193+0000 7fa5e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:57:39.211 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.193+0000 7fa5e1ffb640 10 monclient(hunting): my global_id is 4545 2024-03-29T11:57:39.211 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.193+0000 7fa5e1ffb640 1 -- 172.21.15.98:0/3601937517 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5ec070d30 con 0x7fa5ec1422b0 2024-03-29T11:57:39.211 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.193+0000 7fa5e1ffb640 10 client.?.objecter ms_handle_connect 0x7fa5ec0b17d0 2024-03-29T11:57:39.211 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.193+0000 7fa5e1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:57:39.211 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.193+0000 7fa5e1ffb640 1 -- 172.21.15.98:0/3601937517 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1756759496 0 0) 0x7fa5ec070d30 con 0x7fa5ec1422b0 2024-03-29T11:57:39.211 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.194+0000 7fa5e1ffb640 1 -- 172.21.15.98:0/3601937517 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5ec070770 con 0x7fa5ec1422b0 2024-03-29T11:57:39.211 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.194+0000 7fa5e1ffb640 1 -- 172.21.15.98:0/3601937517 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fa5dc001890 con 0x7fa5ec1422b0 2024-03-29T11:57:39.211 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.194+0000 7fa5e1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:57:39.211 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.194+0000 7fa5e1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:57:39.211 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.194+0000 7fa5e1ffb640 10 monclient(hunting): dump: 2024-03-29T11:57:39.211 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:57:39.211 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:39.212 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:39.212 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:39.212 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:57:39.212 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:39.212 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:57:39.212 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:57:39.212 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:57:39.212 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:39.212 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.194+0000 7fa5e1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:57:39.212 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.194+0000 7fa5e1ffb640 1 -- 172.21.15.98:0/3601937517 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 161288887 0 0) 0x7fa5ec070a50 con 0x7fa5ec0b17d0 2024-03-29T11:57:39.212 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.194+0000 7fa5e1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:57:39.212 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.194+0000 7fa5e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:57:39.212 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.194+0000 7fa5e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:57:39.212 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.194+0000 7fa5e1ffb640 10 monclient(hunting): my global_id is 4546 2024-03-29T11:57:39.212 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.194+0000 7fa5e1ffb640 1 -- 172.21.15.98:0/3601937517 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5ec070d30 con 0x7fa5ec0b17d0 2024-03-29T11:57:39.213 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.195+0000 7fa5e1ffb640 1 -- 172.21.15.98:0/3601937517 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 269496661 0 0) 0x7fa5d0004eb0 con 0x7fa5ec142690 2024-03-29T11:57:39.213 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.195+0000 7fa5e1ffb640 1 -- 172.21.15.98:0/3601937517 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5ec070a50 con 0x7fa5ec142690 2024-03-29T11:57:39.213 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.195+0000 7fa5e1ffb640 1 -- 172.21.15.98:0/3601937517 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 546510761 0 0) 0x7fa5ec070770 con 0x7fa5ec1422b0 2024-03-29T11:57:39.213 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.195+0000 7fa5e1ffb640 1 -- 172.21.15.98:0/3601937517 >> v1:172.21.15.98:6790/0 conn(0x7fa5ec142690 legacy=0x7fa5ec06f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:39.213 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.195+0000 7fa5e1ffb640 1 -- 172.21.15.98:0/3601937517 >> v1:172.21.15.121:6789/0 conn(0x7fa5ec0b17d0 legacy=0x7fa5ec13bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:39.213 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.196+0000 7fa5e1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:57:39.213 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.196+0000 7fa5e1ffb640 1 monclient: found mon.a 2024-03-29T11:57:39.213 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.196+0000 7fa5e1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:57:39.213 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.196+0000 7fa5e1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:57:39.213 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.196+0000 7fa5e1ffb640 1 -- 172.21.15.98:0/3601937517 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ec0b0dd0 con 0x7fa5ec1422b0 2024-03-29T11:57:39.213 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.196+0000 7fa5e1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:57:39.213 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.196+0000 7fa5e1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:57:39.213 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.196+0000 7fa5e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:57:39.213 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.196+0000 7fa5f3674640 5 monclient: authenticate success, global_id 4545 2024-03-29T11:57:39.214 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.196+0000 7fa5f3674640 10 monclient: _renew_subs 2024-03-29T11:57:39.214 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.196+0000 7fa5f3674640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:57:39.214 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.196+0000 7fa5f3674640 1 -- 172.21.15.98:0/3601937517 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5ec0b0940 con 0x7fa5ec1422b0 2024-03-29T11:57:39.214 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.196+0000 7fa5f3674640 10 client.4545.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:57:39.214 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.196+0000 7fa5f3674640 10 monclient: _renew_subs 2024-03-29T11:57:39.214 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.196+0000 7fa5f3674640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:57:39.214 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.196+0000 7fa5f3674640 1 -- 172.21.15.98:0/3601937517 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5ec071980 con 0x7fa5ec1422b0 2024-03-29T11:57:39.214 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.196+0000 7fa5f3674640 1 librados: init done 2024-03-29T11:57:39.214 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.197+0000 7fa5e1ffb640 1 -- 172.21.15.98:0/3601937517 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5dc001d40 con 0x7fa5ec1422b0 2024-03-29T11:57:39.214 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.197+0000 7fa5e1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:57:39.214 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.197+0000 7fa5e1ffb640 1 -- 172.21.15.98:0/3601937517 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fa5dc004980 con 0x7fa5ec1422b0 2024-03-29T11:57:39.214 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.197+0000 7fa5e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:57:39.214 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.197+0000 7fa5e1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:57:39.214 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.197+0000 7fa5e1ffb640 10 monclient: dump: 2024-03-29T11:57:39.214 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:57:39.215 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:39.215 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:39.215 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:39.215 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:57:39.215 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:39.215 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:57:39.215 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:57:39.215 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:57:39.215 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:39.215 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.197+0000 7fa5e37fe640 4 set_mon_vals no callback set 2024-03-29T11:57:39.215 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.198+0000 7fa5f3674640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:57:39.215 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.198+0000 7fa5f3674640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:57:39.215 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.199+0000 7fa5f3674640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:57:39.215 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.199+0000 7fa5f3674640 1 -- 172.21.15.98:0/3601937517 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5cc005080 con 0x7fa5ec1422b0 2024-03-29T11:57:39.215 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.199+0000 7fa5e1ffb640 1 -- 172.21.15.98:0/3601937517 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7fa5dc004ae0 con 0x7fa5ec1422b0 2024-03-29T11:57:39.216 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.200+0000 7fa5f1410640 10 monclient: get_auth_request con 0x7fa5d006ddd0 auth_method 0 2024-03-29T11:57:39.216 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.200+0000 7fa5e1ffb640 1 -- 172.21.15.98:0/3601937517 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7fa5dc084990 con 0x7fa5ec1422b0 2024-03-29T11:57:39.216 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.200+0000 7fa5e1ffb640 10 client.4545.objecter ms_dispatch 0x7fa5ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:57:39.216 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.200+0000 7fa5e1ffb640 3 client.4545.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:57:39.216 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.200+0000 7fa5e1ffb640 3 client.4545.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:57:39.217 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.200+0000 7fa5e1ffb640 20 client.4545.objecter dump_active .. 0 homeless 2024-03-29T11:57:39.217 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.200+0000 7fa5e1ffb640 10 client.4545.objecter ms_handle_connect 0x7fa5d006ddd0 2024-03-29T11:57:39.217 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.207+0000 7fa5e1ffb640 1 -- 172.21.15.98:0/3601937517 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa5dc0500a0 con 0x7fa5ec1422b0 2024-03-29T11:57:39.217 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.208+0000 7fa5e1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:57:39.217 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.208+0000 7fa5e1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:57:39.596 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.588+0000 7fa5f3674640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:57:39.596 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.588+0000 7fa5f3674640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:57:39.596 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.588+0000 7fa5f3674640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:57:39.596 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.588+0000 7fa5f3674640 1 -- 172.21.15.98:0/3601937517 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5cc005330 con 0x7fa5ec1422b0 2024-03-29T11:57:39.597 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.590+0000 7fa5e1ffb640 1 -- 172.21.15.98:0/3601937517 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7fa5dc0521a0 con 0x7fa5ec1422b0 2024-03-29T11:57:39.598 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.590+0000 7fa5e1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:57:39.598 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.590+0000 7fa5e1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:57:39.598 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:57:39.598 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:57:39.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.594+0000 7fa5f3674640 10 librados: watch_flush enter 2024-03-29T11:57:39.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.594+0000 7fa5f3674640 10 librados: watch_flush exit 2024-03-29T11:57:39.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.594+0000 7fa5f3674640 20 client.4545.objecter shutdown clearing up homeless session... 2024-03-29T11:57:39.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.594+0000 7fa5f3674640 10 client.4545.objecter successfully canceled tick 2024-03-29T11:57:39.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.594+0000 7fa5f3674640 1 -- 172.21.15.98:0/3601937517 >> v1:172.21.15.98:6816/4065186974 conn(0x7fa5d006ddd0 legacy=0x7fa5d0070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:39.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.594+0000 7fa5f3674640 10 monclient: shutdown 2024-03-29T11:57:39.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.594+0000 7fa5f3674640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:57:39.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.594+0000 7fa5f3674640 1 -- 172.21.15.98:0/3601937517 >> v1:172.21.15.98:6789/0 conn(0x7fa5ec1422b0 legacy=0x7fa5ec06ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:39.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.594+0000 7fa5f3674640 1 -- 172.21.15.98:0/3601937517 shutdown_connections 2024-03-29T11:57:39.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.594+0000 7fa5f3674640 1 -- 172.21.15.98:0/3601937517 >> 172.21.15.98:0/3601937517 conn(0x7fa5ec066a00 msgr2=0x7fa5ec0b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:57:39.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.594+0000 7fa5f3674640 1 -- 172.21.15.98:0/3601937517 shutdown_connections 2024-03-29T11:57:39.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.594+0000 7fa5f3674640 1 -- 172.21.15.98:0/3601937517 wait complete. 2024-03-29T11:57:39.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:39.594+0000 7fa5f3674640 1 librados: shutdown 2024-03-29T11:57:39.617 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:57:46.424 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:57:46.574 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.566+0000 7f63359cf640 10 monclient: get_monmap_and_config 2024-03-29T11:57:46.575 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.566+0000 7f63359cf640 10 monclient: build_initial_monmap 2024-03-29T11:57:46.575 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.566+0000 7f63359cf640 10 monclient: monmap: 2024-03-29T11:57:46.575 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:57:46.575 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:46.575 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:57:46.567848+0000 2024-03-29T11:57:46.575 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:57:46.567848+0000 2024-03-29T11:57:46.575 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:57:46.575 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:46.575 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:57:46.575 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:57:46.575 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:57:46.576 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:46.576 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.567+0000 7f63359cf640 1 Processor -- start 2024-03-29T11:57:46.576 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.567+0000 7f63359cf640 1 -- start start 2024-03-29T11:57:46.576 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.567+0000 7f63359cf640 10 monclient: init 2024-03-29T11:57:46.576 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.568+0000 7f63359cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:57:46.576 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.568+0000 7f63359cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:57:46.581 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.568+0000 7f63359cf640 10 monclient(hunting): picked mon.noname-c con 0x7f63301457f0 addr v1:172.21.15.121:6789/0 2024-03-29T11:57:46.581 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.568+0000 7f63359cf640 10 monclient(hunting): picked mon.noname-a con 0x7f633013f650 addr v1:172.21.15.98:6789/0 2024-03-29T11:57:46.581 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.568+0000 7f63359cf640 10 monclient(hunting): picked mon.noname-b con 0x7f63300a9d70 addr v1:172.21.15.98:6790/0 2024-03-29T11:57:46.581 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.568+0000 7f63359cf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63300aaef0 con 0x7f633013f650 2024-03-29T11:57:46.581 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.568+0000 7f63359cf640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6330145bd0 con 0x7f63301457f0 2024-03-29T11:57:46.581 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.568+0000 7f63359cf640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6330145eb0 con 0x7f63300a9d70 2024-03-29T11:57:46.581 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.568+0000 7f63359cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:57:46.581 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.568+0000 7f63359cf640 10 monclient(hunting): authenticate will time out at 1373.653958s 2024-03-29T11:57:46.581 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.571+0000 7f632f7fe640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f63300a9d70 0x7f633014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:37427/0 (socket says 172.21.15.98:37427) 2024-03-29T11:57:46.581 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.571+0000 7f632f7fe640 1 -- 172.21.15.98:0/3976394354 learned_addr learned my addr 172.21.15.98:0/3976394354 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:57:46.581 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.572+0000 7f632dffb640 1 -- 172.21.15.98:0/3976394354 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2841952314 0 0) 0x7f6330145eb0 con 0x7f63300a9d70 2024-03-29T11:57:46.581 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.572+0000 7f632dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:57:46.581 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.572+0000 7f632dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:57:46.582 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.572+0000 7f632dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:57:46.582 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.572+0000 7f632dffb640 10 monclient(hunting): my global_id is 4592 2024-03-29T11:57:46.582 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.572+0000 7f632dffb640 1 -- 172.21.15.98:0/3976394354 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6318003520 con 0x7f63300a9d70 2024-03-29T11:57:46.582 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.572+0000 7f632dffb640 1 -- 172.21.15.98:0/3976394354 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3037982255 0 0) 0x7f6318003520 con 0x7f63300a9d70 2024-03-29T11:57:46.582 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.572+0000 7f632dffb640 1 -- 172.21.15.98:0/3976394354 >> v1:172.21.15.121:6789/0 conn(0x7f63301457f0 legacy=0x7f633013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:57:46.582 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.572+0000 7f632dffb640 1 -- 172.21.15.98:0/3976394354 >> v1:172.21.15.98:6789/0 conn(0x7f633013f650 legacy=0x7f63300b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:57:46.583 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.575+0000 7f632dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:57:46.583 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.575+0000 7f632dffb640 1 monclient: found mon.noname-b 2024-03-29T11:57:46.583 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.575+0000 7f632dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:57:46.583 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.575+0000 7f632dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:57:46.583 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.575+0000 7f632dffb640 1 -- 172.21.15.98:0/3976394354 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6330146340 con 0x7f63300a9d70 2024-03-29T11:57:46.583 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.575+0000 7f632dffb640 10 monclient: _finish_auth 0 2024-03-29T11:57:46.583 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.575+0000 7f632dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:57:46.583 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.575+0000 7f632dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:57:46.583 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.575+0000 7f632dffb640 1 -- 172.21.15.98:0/3976394354 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f6324001670 con 0x7f63300a9d70 2024-03-29T11:57:46.584 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.575+0000 7f632dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:57:46.584 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.575+0000 7f632dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:57:46.584 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.575+0000 7f632dffb640 10 monclient: dump: 2024-03-29T11:57:46.584 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:57:46.584 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:46.584 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:46.584 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:46.584 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:57:46.584 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:46.584 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:57:46.584 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:57:46.584 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:57:46.584 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:46.585 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.576+0000 7f63359cf640 5 monclient: authenticate success, global_id 4592 2024-03-29T11:57:46.585 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.576+0000 7f63359cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:57:46.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.577+0000 7f632dffb640 1 -- 172.21.15.98:0/3976394354 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6324001840 con 0x7f63300a9d70 2024-03-29T11:57:46.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.577+0000 7f632dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:57:46.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.577+0000 7f632dffb640 1 -- 172.21.15.98:0/3976394354 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f6324001c90 con 0x7f63300a9d70 2024-03-29T11:57:46.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.577+0000 7f632dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:57:46.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.577+0000 7f632dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:57:46.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.577+0000 7f632dffb640 10 monclient: dump: 2024-03-29T11:57:46.588 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:57:46.588 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:46.588 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:46.588 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:46.588 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:57:46.588 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:46.589 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:57:46.589 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:57:46.589 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:57:46.589 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:46.589 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.579+0000 7f63359cf640 10 monclient: get_monmap_and_config success 2024-03-29T11:57:46.589 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.579+0000 7f63359cf640 4 set_mon_vals no callback set 2024-03-29T11:57:46.589 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.580+0000 7f63359cf640 10 monclient: shutdown 2024-03-29T11:57:46.589 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.580+0000 7f63359cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:57:46.589 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.580+0000 7f63359cf640 1 -- 172.21.15.98:0/3976394354 >> v1:172.21.15.98:6790/0 conn(0x7f63300a9d70 legacy=0x7f633014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:46.590 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.583+0000 7f63359cf640 1 -- 172.21.15.98:0/3976394354 shutdown_connections 2024-03-29T11:57:46.590 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.583+0000 7f63359cf640 1 -- 172.21.15.98:0/3976394354 >> 172.21.15.98:0/3976394354 conn(0x7f6330066a00 msgr2=0x7f6330066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:57:46.591 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.583+0000 7f63359cf640 1 -- 172.21.15.98:0/3976394354 shutdown_connections 2024-03-29T11:57:46.593 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.584+0000 7f63359cf640 1 -- 172.21.15.98:0/3976394354 wait complete. 2024-03-29T11:57:46.593 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.584+0000 7f63359cf640 10 monclient: build_initial_monmap 2024-03-29T11:57:46.593 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.584+0000 7f63359cf640 10 monclient: monmap: 2024-03-29T11:57:46.593 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:57:46.593 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:57:46.593 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:57:46.593 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:57:46.594 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:57:46.594 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:46.594 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:57:46.594 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:57:46.594 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:57:46.594 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:46.594 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.585+0000 7f63359cf640 1 librados: starting msgr at 2024-03-29T11:57:46.594 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.585+0000 7f63359cf640 1 librados: starting objecter 2024-03-29T11:57:46.594 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.585+0000 7f63359cf640 1 Processor -- start 2024-03-29T11:57:46.595 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.586+0000 7f63359cf640 1 -- start start 2024-03-29T11:57:46.596 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.586+0000 7f63359cf640 1 librados: setting wanted keys 2024-03-29T11:57:46.596 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.586+0000 7f63359cf640 1 librados: calling monclient init 2024-03-29T11:57:46.596 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.586+0000 7f63359cf640 10 monclient: init 2024-03-29T11:57:46.596 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.587+0000 7f63359cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:57:46.596 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.587+0000 7f63359cf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:57:46.596 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.588+0000 7f63359cf640 10 monclient(hunting): picked mon.noname-c con 0x7f63300a9d70 addr v1:172.21.15.98:6790/0 2024-03-29T11:57:46.596 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.588+0000 7f63359cf640 10 monclient(hunting): picked mon.noname-b con 0x7f633013f650 addr v1:172.21.15.121:6789/0 2024-03-29T11:57:46.596 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.589+0000 7f63359cf640 10 monclient(hunting): picked mon.noname-a con 0x7f63301457f0 addr v1:172.21.15.98:6789/0 2024-03-29T11:57:46.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.591+0000 7f63359cf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6330140ff0 con 0x7f63301457f0 2024-03-29T11:57:46.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.591+0000 7f63359cf640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6330141db0 con 0x7f633013f650 2024-03-29T11:57:46.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.591+0000 7f63359cf640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6330142090 con 0x7f63300a9d70 2024-03-29T11:57:46.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.591+0000 7f63359cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:57:46.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.591+0000 7f63359cf640 10 monclient(hunting): authenticate will time out at 1373.676866s 2024-03-29T11:57:46.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.592+0000 7f632f7fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f63301457f0 0x7f63301e63c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:59913/0 (socket says 172.21.15.98:59913) 2024-03-29T11:57:46.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.592+0000 7f632f7fe640 1 -- 172.21.15.98:0/2016478248 learned_addr learned my addr 172.21.15.98:0/2016478248 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:57:46.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.592+0000 7f6317fff640 10 client.?.objecter ms_handle_connect 0x7f63301457f0 2024-03-29T11:57:46.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.592+0000 7f6317fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:57:46.607 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.594+0000 7f6317fff640 1 -- 172.21.15.98:0/2016478248 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 906833440 0 0) 0x7f6330140ff0 con 0x7f63301457f0 2024-03-29T11:57:46.607 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.594+0000 7f6317fff640 10 monclient(hunting): _negotiate 2024-03-29T11:57:46.607 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.594+0000 7f6317fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:57:46.607 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.594+0000 7f6317fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:57:46.607 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.594+0000 7f6317fff640 10 monclient(hunting): my global_id is 4548 2024-03-29T11:57:46.607 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.594+0000 7f6317fff640 1 -- 172.21.15.98:0/2016478248 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f631c004eb0 con 0x7f63301457f0 2024-03-29T11:57:46.607 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.594+0000 7f6317fff640 10 client.?.objecter ms_handle_connect 0x7f633013f650 2024-03-29T11:57:46.607 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.595+0000 7f6317fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:57:46.607 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.595+0000 7f6317fff640 10 client.?.objecter ms_handle_connect 0x7f63300a9d70 2024-03-29T11:57:46.607 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.595+0000 7f6317fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:57:46.607 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.595+0000 7f6317fff640 1 -- 172.21.15.98:0/2016478248 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4260313581 0 0) 0x7f6330142090 con 0x7f63300a9d70 2024-03-29T11:57:46.608 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.595+0000 7f6317fff640 10 monclient(hunting): _negotiate 2024-03-29T11:57:46.608 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.595+0000 7f6317fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:57:46.608 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.595+0000 7f6317fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:57:46.608 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.595+0000 7f6317fff640 10 monclient(hunting): my global_id is 4595 2024-03-29T11:57:46.608 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.595+0000 7f6317fff640 1 -- 172.21.15.98:0/2016478248 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6330140ff0 con 0x7f63300a9d70 2024-03-29T11:57:46.608 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.595+0000 7f6317fff640 1 -- 172.21.15.98:0/2016478248 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4230269722 0 0) 0x7f631c004eb0 con 0x7f63301457f0 2024-03-29T11:57:46.608 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.595+0000 7f6317fff640 1 -- 172.21.15.98:0/2016478248 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6330142090 con 0x7f63301457f0 2024-03-29T11:57:46.608 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.595+0000 7f6317fff640 1 -- 172.21.15.98:0/2016478248 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2285555924 0 0) 0x7f6330141db0 con 0x7f633013f650 2024-03-29T11:57:46.608 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.595+0000 7f6317fff640 10 monclient(hunting): _negotiate 2024-03-29T11:57:46.608 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.595+0000 7f6317fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:57:46.608 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.595+0000 7f6317fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:57:46.608 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.595+0000 7f6317fff640 10 monclient(hunting): my global_id is 4549 2024-03-29T11:57:46.608 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.595+0000 7f6317fff640 1 -- 172.21.15.98:0/2016478248 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f631c004eb0 con 0x7f633013f650 2024-03-29T11:57:46.609 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.595+0000 7f6317fff640 1 -- 172.21.15.98:0/2016478248 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f6324001c90 con 0x7f63301457f0 2024-03-29T11:57:46.609 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.595+0000 7f6317fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:57:46.609 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.595+0000 7f6317fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:57:46.609 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.595+0000 7f6317fff640 10 monclient(hunting): dump: 2024-03-29T11:57:46.609 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:57:46.609 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:46.609 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:46.609 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:46.609 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:57:46.609 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:46.609 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:57:46.609 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:57:46.609 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:57:46.609 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:46.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.595+0000 7f6317fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:57:46.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.595+0000 7f6317fff640 1 -- 172.21.15.98:0/2016478248 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2963837153 0 0) 0x7f6330142090 con 0x7f63301457f0 2024-03-29T11:57:46.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.596+0000 7f6317fff640 1 -- 172.21.15.98:0/2016478248 >> v1:172.21.15.98:6790/0 conn(0x7f63300a9d70 legacy=0x7f633013fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:46.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.596+0000 7f6317fff640 1 -- 172.21.15.98:0/2016478248 >> v1:172.21.15.121:6789/0 conn(0x7f633013f650 legacy=0x7f633013a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:46.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.596+0000 7f6317fff640 10 monclient: _finish_hunting 0 2024-03-29T11:57:46.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.596+0000 7f6317fff640 1 monclient: found mon.a 2024-03-29T11:57:46.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.596+0000 7f6317fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:57:46.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.596+0000 7f6317fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:57:46.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.596+0000 7f6317fff640 1 -- 172.21.15.98:0/2016478248 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6330142370 con 0x7f63301457f0 2024-03-29T11:57:46.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.596+0000 7f6317fff640 10 monclient: _finish_auth 0 2024-03-29T11:57:46.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.596+0000 7f6317fff640 10 monclient: _check_auth_tickets 2024-03-29T11:57:46.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.596+0000 7f6317fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:57:46.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.596+0000 7f6317fff640 1 -- 172.21.15.98:0/2016478248 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6324004bf0 con 0x7f63301457f0 2024-03-29T11:57:46.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.596+0000 7f63359cf640 5 monclient: authenticate success, global_id 4548 2024-03-29T11:57:46.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.596+0000 7f63359cf640 10 monclient: _renew_subs 2024-03-29T11:57:46.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.596+0000 7f63359cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:57:46.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.596+0000 7f63359cf640 1 -- 172.21.15.98:0/2016478248 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6330135c00 con 0x7f63301457f0 2024-03-29T11:57:46.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.597+0000 7f63359cf640 10 client.4548.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:57:46.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.597+0000 7f63359cf640 10 monclient: _renew_subs 2024-03-29T11:57:46.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.597+0000 7f63359cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:57:46.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.597+0000 7f63359cf640 1 -- 172.21.15.98:0/2016478248 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f63301411f0 con 0x7f63301457f0 2024-03-29T11:57:46.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.597+0000 7f63359cf640 1 librados: init done 2024-03-29T11:57:46.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.597+0000 7f6317fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:57:46.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.597+0000 7f6317fff640 1 -- 172.21.15.98:0/2016478248 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f6324004f00 con 0x7f63301457f0 2024-03-29T11:57:46.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.597+0000 7f6317fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:57:46.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.597+0000 7f6317fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:57:46.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.597+0000 7f6317fff640 10 monclient: dump: 2024-03-29T11:57:46.611 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:57:46.612 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:46.612 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:46.612 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:46.612 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:57:46.612 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:46.612 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:57:46.612 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:57:46.612 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:57:46.612 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:46.612 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.598+0000 7f632d7fa640 4 set_mon_vals no callback set 2024-03-29T11:57:46.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.600+0000 7f63359cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:57:46.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.600+0000 7f63359cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:57:46.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.600+0000 7f63359cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:57:46.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.600+0000 7f63359cf640 1 -- 172.21.15.98:0/2016478248 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6310005080 con 0x7f63301457f0 2024-03-29T11:57:46.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.601+0000 7f6317fff640 1 -- 172.21.15.98:0/2016478248 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f632401aed0 con 0x7f63301457f0 2024-03-29T11:57:46.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.602+0000 7f632effd640 10 monclient: get_auth_request con 0x7f631c06da30 auth_method 0 2024-03-29T11:57:46.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.602+0000 7f6317fff640 1 -- 172.21.15.98:0/2016478248 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f6324084c20 con 0x7f63301457f0 2024-03-29T11:57:46.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.602+0000 7f6317fff640 10 client.4548.objecter ms_dispatch 0x7f6330000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:57:46.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.602+0000 7f6317fff640 3 client.4548.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:57:46.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.602+0000 7f6317fff640 3 client.4548.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:57:46.615 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.602+0000 7f6317fff640 20 client.4548.objecter dump_active .. 0 homeless 2024-03-29T11:57:46.615 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.602+0000 7f6317fff640 10 client.4548.objecter ms_handle_connect 0x7f631c06da30 2024-03-29T11:57:46.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.608+0000 7f6317fff640 1 -- 172.21.15.98:0/2016478248 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6324050500 con 0x7f63301457f0 2024-03-29T11:57:46.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.608+0000 7f6317fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:57:46.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.608+0000 7f6317fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:57:46.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.927+0000 7f63359cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:57:46.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.927+0000 7f63359cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:57:46.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.927+0000 7f63359cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:57:46.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.927+0000 7f63359cf640 1 -- 172.21.15.98:0/2016478248 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6310005330 con 0x7f63301457f0 2024-03-29T11:57:46.940 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.932+0000 7f6317fff640 1 -- 172.21.15.98:0/2016478248 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f6324052430 con 0x7f63301457f0 2024-03-29T11:57:46.940 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.932+0000 7f6317fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:57:46.940 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.932+0000 7f6317fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:57:46.941 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:57:46.941 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:57:46.943 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.936+0000 7f63359cf640 10 librados: watch_flush enter 2024-03-29T11:57:46.943 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.936+0000 7f63359cf640 10 librados: watch_flush exit 2024-03-29T11:57:46.943 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.936+0000 7f63359cf640 20 client.4548.objecter shutdown clearing up homeless session... 2024-03-29T11:57:46.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.936+0000 7f63359cf640 10 client.4548.objecter successfully canceled tick 2024-03-29T11:57:46.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.936+0000 7f63359cf640 1 -- 172.21.15.98:0/2016478248 >> v1:172.21.15.98:6816/4065186974 conn(0x7f631c06da30 legacy=0x7f631c06fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:46.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.936+0000 7f63359cf640 10 monclient: shutdown 2024-03-29T11:57:46.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.936+0000 7f63359cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:57:46.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.936+0000 7f63359cf640 1 -- 172.21.15.98:0/2016478248 >> v1:172.21.15.98:6789/0 conn(0x7f63301457f0 legacy=0x7f63301e63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:46.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.937+0000 7f63359cf640 1 -- 172.21.15.98:0/2016478248 shutdown_connections 2024-03-29T11:57:46.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.937+0000 7f63359cf640 1 -- 172.21.15.98:0/2016478248 >> 172.21.15.98:0/2016478248 conn(0x7f6330066a00 msgr2=0x7f63300b4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:57:46.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.939+0000 7f63359cf640 1 -- 172.21.15.98:0/2016478248 shutdown_connections 2024-03-29T11:57:46.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.941+0000 7f63359cf640 1 -- 172.21.15.98:0/2016478248 wait complete. 2024-03-29T11:57:46.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:46.942+0000 7f63359cf640 1 librados: shutdown 2024-03-29T11:57:46.969 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:57:53.775 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:57:53.918 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.909+0000 7f6390e55640 10 monclient: get_monmap_and_config 2024-03-29T11:57:53.918 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.909+0000 7f6390e55640 10 monclient: build_initial_monmap 2024-03-29T11:57:53.918 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.909+0000 7f6390e55640 10 monclient: monmap: 2024-03-29T11:57:53.918 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:57:53.918 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:53.918 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:57:53.910841+0000 2024-03-29T11:57:53.918 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:57:53.910841+0000 2024-03-29T11:57:53.918 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:57:53.919 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:53.919 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:57:53.919 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:57:53.919 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:57:53.919 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:53.919 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.912+0000 7f6390e55640 1 Processor -- start 2024-03-29T11:57:53.922 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.912+0000 7f6390e55640 1 -- start start 2024-03-29T11:57:53.922 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.912+0000 7f6390e55640 10 monclient: init 2024-03-29T11:57:53.922 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.913+0000 7f6390e55640 10 monclient: _reopen_session rank -1 2024-03-29T11:57:53.922 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.913+0000 7f6390e55640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:57:53.922 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.913+0000 7f6390e55640 10 monclient(hunting): picked mon.noname-a con 0x7f638c0a9ac0 addr v1:172.21.15.98:6789/0 2024-03-29T11:57:53.928 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.916+0000 7f6390e55640 10 monclient(hunting): picked mon.noname-c con 0x7f638c13ee90 addr v1:172.21.15.121:6789/0 2024-03-29T11:57:53.928 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.916+0000 7f6390e55640 10 monclient(hunting): picked mon.noname-b con 0x7f638c0af900 addr v1:172.21.15.98:6790/0 2024-03-29T11:57:53.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.916+0000 7f638a59c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f638c0a9ac0 0x7f638c0adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44719/0 (socket says 172.21.15.98:44719) 2024-03-29T11:57:53.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.916+0000 7f638a59c640 1 -- 172.21.15.98:0/804513293 learned_addr learned my addr 172.21.15.98:0/804513293 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:57:53.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.916+0000 7f6390e55640 1 -- 172.21.15.98:0/804513293 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f638c0afd10 con 0x7f638c0a9ac0 2024-03-29T11:57:53.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.916+0000 7f6390e55640 1 -- 172.21.15.98:0/804513293 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f638c0af6f0 con 0x7f638c13ee90 2024-03-29T11:57:53.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.917+0000 7f6390e55640 1 -- 172.21.15.98:0/804513293 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f638c0aac80 con 0x7f638c0af900 2024-03-29T11:57:53.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.917+0000 7f6390e55640 10 monclient(hunting): _renew_subs 2024-03-29T11:57:53.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.917+0000 7f6390e55640 10 monclient(hunting): authenticate will time out at 1381.002242s 2024-03-29T11:57:53.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.917+0000 7f638959a640 1 -- 172.21.15.98:0/804513293 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1762361214 0 0) 0x7f638c0afd10 con 0x7f638c0a9ac0 2024-03-29T11:57:53.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.917+0000 7f638959a640 10 monclient(hunting): _negotiate 2024-03-29T11:57:53.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.917+0000 7f638959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:57:53.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.917+0000 7f638959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:57:53.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.917+0000 7f638959a640 10 monclient(hunting): my global_id is 4551 2024-03-29T11:57:53.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.917+0000 7f638959a640 1 -- 172.21.15.98:0/804513293 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6374003520 con 0x7f638c0a9ac0 2024-03-29T11:57:53.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.917+0000 7f638959a640 1 -- 172.21.15.98:0/804513293 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3007758829 0 0) 0x7f638c0af6f0 con 0x7f638c13ee90 2024-03-29T11:57:53.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.917+0000 7f638959a640 10 monclient(hunting): _negotiate 2024-03-29T11:57:53.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.917+0000 7f638959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:57:53.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.917+0000 7f638959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:57:53.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.917+0000 7f638959a640 10 monclient(hunting): my global_id is 4552 2024-03-29T11:57:53.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.917+0000 7f638959a640 1 -- 172.21.15.98:0/804513293 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f638c0afd10 con 0x7f638c13ee90 2024-03-29T11:57:53.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.917+0000 7f638959a640 1 -- 172.21.15.98:0/804513293 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2573096709 0 0) 0x7f638c0aac80 con 0x7f638c0af900 2024-03-29T11:57:53.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.918+0000 7f638959a640 10 monclient(hunting): _negotiate 2024-03-29T11:57:53.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.918+0000 7f638959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:57:53.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.918+0000 7f638959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:57:53.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.918+0000 7f638959a640 10 monclient(hunting): my global_id is 4598 2024-03-29T11:57:53.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.918+0000 7f638959a640 1 -- 172.21.15.98:0/804513293 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f638c0af6f0 con 0x7f638c0af900 2024-03-29T11:57:53.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.918+0000 7f638959a640 1 -- 172.21.15.98:0/804513293 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2085406466 0 0) 0x7f638c0afd10 con 0x7f638c13ee90 2024-03-29T11:57:53.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.918+0000 7f638959a640 1 -- 172.21.15.98:0/804513293 >> v1:172.21.15.98:6790/0 conn(0x7f638c0af900 legacy=0x7f638c149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:53.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.918+0000 7f638959a640 1 -- 172.21.15.98:0/804513293 >> v1:172.21.15.98:6789/0 conn(0x7f638c0a9ac0 legacy=0x7f638c0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:53.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.918+0000 7f638959a640 10 monclient: _finish_hunting 0 2024-03-29T11:57:53.931 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.918+0000 7f638959a640 1 monclient: found mon.noname-c 2024-03-29T11:57:53.931 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.918+0000 7f638959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:57:53.931 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.918+0000 7f638959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:57:53.931 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.918+0000 7f638959a640 1 -- 172.21.15.98:0/804513293 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f638c0aaf60 con 0x7f638c13ee90 2024-03-29T11:57:53.931 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.918+0000 7f638959a640 10 monclient: _finish_auth 0 2024-03-29T11:57:53.931 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.918+0000 7f638959a640 10 monclient: _check_auth_tickets 2024-03-29T11:57:53.931 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.919+0000 7f638959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:57:53.931 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.919+0000 7f638959a640 1 -- 172.21.15.98:0/804513293 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f63840019f0 con 0x7f638c13ee90 2024-03-29T11:57:53.931 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.919+0000 7f6390e55640 5 monclient: authenticate success, global_id 4552 2024-03-29T11:57:53.931 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.919+0000 7f6390e55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:57:53.931 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.919+0000 7f638959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:57:53.931 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.919+0000 7f638959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:57:53.931 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.919+0000 7f638959a640 10 monclient: dump: 2024-03-29T11:57:53.931 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:57:53.931 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:53.931 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:53.932 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:53.932 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:57:53.932 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:53.932 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:57:53.932 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:57:53.932 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:57:53.932 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:53.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.919+0000 7f638959a640 1 -- 172.21.15.98:0/804513293 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6384001c10 con 0x7f638c13ee90 2024-03-29T11:57:53.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.919+0000 7f6390e55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:57:53.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.919+0000 7f638959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:57:53.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.919+0000 7f638959a640 1 -- 172.21.15.98:0/804513293 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f6384001fa0 con 0x7f638c13ee90 2024-03-29T11:57:53.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.919+0000 7f6390e55640 10 monclient: get_monmap_and_config success 2024-03-29T11:57:53.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.919+0000 7f6390e55640 4 set_mon_vals no callback set 2024-03-29T11:57:53.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.920+0000 7f6390e55640 10 monclient: shutdown 2024-03-29T11:57:53.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.920+0000 7f638959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:57:53.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.920+0000 7f638959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:57:53.933 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.920+0000 7f638959a640 10 monclient: dump: 2024-03-29T11:57:53.933 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:57:53.933 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:53.933 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:53.933 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:53.933 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:57:53.933 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:53.933 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:57:53.933 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:57:53.933 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:57:53.933 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:53.933 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.920+0000 7f6390e55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:57:53.933 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.920+0000 7f6390e55640 1 -- 172.21.15.98:0/804513293 >> v1:172.21.15.121:6789/0 conn(0x7f638c13ee90 legacy=0x7f638c13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:53.933 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.921+0000 7f6390e55640 1 -- 172.21.15.98:0/804513293 shutdown_connections 2024-03-29T11:57:53.933 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.921+0000 7f6390e55640 1 -- 172.21.15.98:0/804513293 >> 172.21.15.98:0/804513293 conn(0x7f638c0b35e0 msgr2=0x7f638c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:57:53.935 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.921+0000 7f6390e55640 1 -- 172.21.15.98:0/804513293 shutdown_connections 2024-03-29T11:57:53.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.921+0000 7f6390e55640 1 -- 172.21.15.98:0/804513293 wait complete. 2024-03-29T11:57:53.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.923+0000 7f6390e55640 10 monclient: build_initial_monmap 2024-03-29T11:57:53.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.923+0000 7f6390e55640 10 monclient: monmap: 2024-03-29T11:57:53.936 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:57:53.936 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:57:53.936 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:57:53.936 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:57:53.936 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:57:53.936 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:53.936 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:57:53.936 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:57:53.936 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:57:53.936 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:53.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.923+0000 7f6390e55640 1 librados: starting msgr at 2024-03-29T11:57:53.937 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.923+0000 7f6390e55640 1 librados: starting objecter 2024-03-29T11:57:53.937 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.923+0000 7f6390e55640 1 Processor -- start 2024-03-29T11:57:53.937 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.924+0000 7f6390e55640 1 -- start start 2024-03-29T11:57:53.937 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.924+0000 7f6390e55640 1 librados: setting wanted keys 2024-03-29T11:57:53.937 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.924+0000 7f6390e55640 1 librados: calling monclient init 2024-03-29T11:57:53.937 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.924+0000 7f6390e55640 10 monclient: init 2024-03-29T11:57:53.937 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.924+0000 7f6390e55640 10 monclient: _reopen_session rank -1 2024-03-29T11:57:53.937 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.924+0000 7f6390e55640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:57:53.937 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.924+0000 7f6390e55640 10 monclient(hunting): picked mon.noname-c con 0x7f638c0a9ac0 addr v1:172.21.15.98:6790/0 2024-03-29T11:57:53.937 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.924+0000 7f6390e55640 10 monclient(hunting): picked mon.noname-a con 0x7f638c0af900 addr v1:172.21.15.98:6789/0 2024-03-29T11:57:53.937 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.924+0000 7f6390e55640 10 monclient(hunting): picked mon.noname-b con 0x7f638c13ee90 addr v1:172.21.15.121:6789/0 2024-03-29T11:57:53.937 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.924+0000 7f6390e55640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f638c1338c0 con 0x7f638c0af900 2024-03-29T11:57:53.937 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.925+0000 7f6390e55640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f638c133de0 con 0x7f638c13ee90 2024-03-29T11:57:53.937 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.925+0000 7f6390e55640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f638c133f70 con 0x7f638c0a9ac0 2024-03-29T11:57:53.937 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.925+0000 7f6390e55640 10 monclient(hunting): _renew_subs 2024-03-29T11:57:53.937 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.925+0000 7f6390e55640 10 monclient(hunting): authenticate will time out at 1381.010081s 2024-03-29T11:57:53.938 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.925+0000 7f638a59c640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f638c0a9ac0 0x7f638c131930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:40156/0 (socket says 172.21.15.98:40156) 2024-03-29T11:57:53.938 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.925+0000 7f638a59c640 1 -- 172.21.15.98:0/3030956171 learned_addr learned my addr 172.21.15.98:0/3030956171 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:57:53.938 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.925+0000 7f637affd640 10 client.?.objecter ms_handle_connect 0x7f638c0a9ac0 2024-03-29T11:57:53.938 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.925+0000 7f637affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:57:53.938 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.929+0000 7f637affd640 10 client.?.objecter ms_handle_connect 0x7f638c0af900 2024-03-29T11:57:53.938 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.929+0000 7f637affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:57:53.938 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.929+0000 7f637affd640 1 -- 172.21.15.98:0/3030956171 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2336692225 0 0) 0x7f6374003520 con 0x7f638c0a9ac0 2024-03-29T11:57:53.938 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.929+0000 7f637affd640 10 monclient(hunting): _negotiate 2024-03-29T11:57:53.938 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.929+0000 7f637affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:57:53.938 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.929+0000 7f637affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:57:53.938 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.929+0000 7f637affd640 10 monclient(hunting): my global_id is 4601 2024-03-29T11:57:53.939 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.929+0000 7f637affd640 1 -- 172.21.15.98:0/3030956171 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f636c004eb0 con 0x7f638c0a9ac0 2024-03-29T11:57:53.939 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.929+0000 7f637affd640 1 -- 172.21.15.98:0/3030956171 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1953418627 0 0) 0x7f638c1338c0 con 0x7f638c0af900 2024-03-29T11:57:53.939 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.929+0000 7f637affd640 10 monclient(hunting): _negotiate 2024-03-29T11:57:53.939 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.929+0000 7f637affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:57:53.939 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.929+0000 7f637affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:57:53.939 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.929+0000 7f637affd640 10 monclient(hunting): my global_id is 4554 2024-03-29T11:57:53.939 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.929+0000 7f637affd640 1 -- 172.21.15.98:0/3030956171 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6374003520 con 0x7f638c0af900 2024-03-29T11:57:53.940 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.932+0000 7f637affd640 10 client.?.objecter ms_handle_connect 0x7f638c13ee90 2024-03-29T11:57:53.940 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.932+0000 7f637affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:57:53.940 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.932+0000 7f637affd640 1 -- 172.21.15.98:0/3030956171 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2483333588 0 0) 0x7f636c004eb0 con 0x7f638c0a9ac0 2024-03-29T11:57:53.940 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.932+0000 7f637affd640 1 -- 172.21.15.98:0/3030956171 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f638c1338c0 con 0x7f638c0a9ac0 2024-03-29T11:57:53.940 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.932+0000 7f637affd640 1 -- 172.21.15.98:0/3030956171 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1047566527 0 0) 0x7f6374003520 con 0x7f638c0af900 2024-03-29T11:57:53.940 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.932+0000 7f637affd640 1 -- 172.21.15.98:0/3030956171 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f636c004eb0 con 0x7f638c0af900 2024-03-29T11:57:53.940 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.932+0000 7f637affd640 1 -- 172.21.15.98:0/3030956171 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f637c005cb0 con 0x7f638c0a9ac0 2024-03-29T11:57:53.941 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.932+0000 7f637affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:57:53.941 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.932+0000 7f637affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:57:53.941 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.932+0000 7f637affd640 10 monclient(hunting): dump: 2024-03-29T11:57:53.941 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:57:53.941 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:53.941 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:53.941 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:53.941 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:57:53.941 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:53.941 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:57:53.941 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:57:53.941 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:57:53.941 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:53.941 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.932+0000 7f637affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:57:53.941 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.932+0000 7f637affd640 1 -- 172.21.15.98:0/3030956171 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f6384001f20 con 0x7f638c0af900 2024-03-29T11:57:53.941 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.932+0000 7f637affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:57:53.942 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.932+0000 7f637affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:57:53.942 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.932+0000 7f637affd640 10 monclient(hunting): dump: 2024-03-29T11:57:53.942 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:57:53.942 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:53.942 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:53.942 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:53.942 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:57:53.942 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:53.942 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:57:53.942 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:57:53.942 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:57:53.942 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:53.942 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.932+0000 7f637affd640 1 -- 172.21.15.98:0/3030956171 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3608918144 0 0) 0x7f638c1338c0 con 0x7f638c0a9ac0 2024-03-29T11:57:53.942 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.932+0000 7f637affd640 1 -- 172.21.15.98:0/3030956171 >> v1:172.21.15.121:6789/0 conn(0x7f638c13ee90 legacy=0x7f638c1de500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:53.943 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.932+0000 7f637affd640 1 -- 172.21.15.98:0/3030956171 >> v1:172.21.15.98:6789/0 conn(0x7f638c0af900 legacy=0x7f638c1d2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:53.943 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.932+0000 7f637affd640 10 monclient: _finish_hunting 0 2024-03-29T11:57:53.943 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.932+0000 7f637affd640 1 monclient: found mon.c 2024-03-29T11:57:53.943 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.932+0000 7f637affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:57:53.943 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.932+0000 7f637affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:57:53.943 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.932+0000 7f637affd640 1 -- 172.21.15.98:0/3030956171 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f638c1323c0 con 0x7f638c0a9ac0 2024-03-29T11:57:53.943 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.932+0000 7f637affd640 10 monclient: _finish_auth 0 2024-03-29T11:57:53.943 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.932+0000 7f637affd640 10 monclient: _check_auth_tickets 2024-03-29T11:57:53.943 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.932+0000 7f637affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:57:53.943 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.932+0000 7f6390e55640 5 monclient: authenticate success, global_id 4601 2024-03-29T11:57:53.943 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.932+0000 7f6390e55640 10 monclient: _renew_subs 2024-03-29T11:57:53.943 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.932+0000 7f6390e55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:57:53.943 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.932+0000 7f6390e55640 1 -- 172.21.15.98:0/3030956171 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f638c132040 con 0x7f638c0a9ac0 2024-03-29T11:57:53.943 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.932+0000 7f6390e55640 10 client.4601.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:57:53.943 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.932+0000 7f6390e55640 10 monclient: _renew_subs 2024-03-29T11:57:53.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.932+0000 7f6390e55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:57:53.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.932+0000 7f6390e55640 1 -- 172.21.15.98:0/3030956171 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f638c132a80 con 0x7f638c0a9ac0 2024-03-29T11:57:53.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.932+0000 7f637affd640 1 -- 172.21.15.98:0/3030956171 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f637c005fd0 con 0x7f638c0a9ac0 2024-03-29T11:57:53.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.932+0000 7f6390e55640 1 librados: init done 2024-03-29T11:57:53.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.933+0000 7f637affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:57:53.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.933+0000 7f637affd640 1 -- 172.21.15.98:0/3030956171 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f637c0016d0 con 0x7f638c0a9ac0 2024-03-29T11:57:53.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.933+0000 7f637affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:57:53.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.933+0000 7f637affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:57:53.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.933+0000 7f637affd640 10 monclient: dump: 2024-03-29T11:57:53.944 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:57:53.944 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:57:53.944 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:53.944 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:57:53.944 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:57:53.944 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:57:53.944 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:57:53.944 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:57:53.945 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:57:53.945 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:57:53.945 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.933+0000 7f6388d99640 4 set_mon_vals no callback set 2024-03-29T11:57:53.945 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.935+0000 7f637affd640 1 -- 172.21.15.98:0/3030956171 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f637c001830 con 0x7f638c0a9ac0 2024-03-29T11:57:53.945 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.936+0000 7f6390e55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:57:53.945 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.936+0000 7f6390e55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:57:53.945 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.936+0000 7f6390e55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:57:53.945 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.936+0000 7f6390e55640 1 -- 172.21.15.98:0/3030956171 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6370005080 con 0x7f638c0a9ac0 2024-03-29T11:57:53.946 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.938+0000 7f637affd640 1 -- 172.21.15.98:0/3030956171 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f637c088cc0 con 0x7f638c0a9ac0 2024-03-29T11:57:53.946 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.938+0000 7f637affd640 10 client.4601.objecter ms_dispatch 0x7f638c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:57:53.946 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.938+0000 7f637affd640 3 client.4601.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:57:53.946 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.938+0000 7f637affd640 3 client.4601.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:57:53.946 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.938+0000 7f637affd640 20 client.4601.objecter dump_active .. 0 homeless 2024-03-29T11:57:53.946 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.938+0000 7f6389d9b640 10 monclient: get_auth_request con 0x7f636c06dec0 auth_method 0 2024-03-29T11:57:53.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.939+0000 7f637affd640 10 client.4601.objecter ms_handle_connect 0x7f636c06dec0 2024-03-29T11:57:53.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.946+0000 7f637affd640 1 -- 172.21.15.98:0/3030956171 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f637c0545a0 con 0x7f638c0a9ac0 2024-03-29T11:57:53.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.946+0000 7f637affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:57:53.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:53.946+0000 7f637affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:57:54.278 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:54.271+0000 7f6390e55640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:57:54.278 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:54.271+0000 7f6390e55640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:57:54.278 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:54.271+0000 7f6390e55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:57:54.278 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:54.271+0000 7f6390e55640 1 -- 172.21.15.98:0/3030956171 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6370005330 con 0x7f638c0a9ac0 2024-03-29T11:57:54.280 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:54.273+0000 7f637affd640 1 -- 172.21.15.98:0/3030956171 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f637c0564d0 con 0x7f638c0a9ac0 2024-03-29T11:57:54.280 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:54.273+0000 7f637affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:57:54.281 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:54.273+0000 7f637affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:57:54.282 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:57:54.282 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:57:54.285 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:54.277+0000 7f6390e55640 10 librados: watch_flush enter 2024-03-29T11:57:54.285 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:54.278+0000 7f6390e55640 10 librados: watch_flush exit 2024-03-29T11:57:54.285 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:54.278+0000 7f6390e55640 20 client.4601.objecter shutdown clearing up homeless session... 2024-03-29T11:57:54.285 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:54.278+0000 7f6390e55640 10 client.4601.objecter successfully canceled tick 2024-03-29T11:57:54.286 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:54.279+0000 7f6390e55640 1 -- 172.21.15.98:0/3030956171 >> v1:172.21.15.98:6816/4065186974 conn(0x7f636c06dec0 legacy=0x7f636c070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:54.286 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:54.279+0000 7f6390e55640 10 monclient: shutdown 2024-03-29T11:57:54.286 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:54.279+0000 7f6390e55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:57:54.287 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:54.279+0000 7f6390e55640 1 -- 172.21.15.98:0/3030956171 >> v1:172.21.15.98:6790/0 conn(0x7f638c0a9ac0 legacy=0x7f638c131930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:57:54.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:54.280+0000 7f6390e55640 1 -- 172.21.15.98:0/3030956171 shutdown_connections 2024-03-29T11:57:54.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:54.280+0000 7f6390e55640 1 -- 172.21.15.98:0/3030956171 >> 172.21.15.98:0/3030956171 conn(0x7f638c0b35e0 msgr2=0x7f638c0ad060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:57:54.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:54.280+0000 7f6390e55640 1 -- 172.21.15.98:0/3030956171 shutdown_connections 2024-03-29T11:57:54.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:54.280+0000 7f6390e55640 1 -- 172.21.15.98:0/3030956171 wait complete. 2024-03-29T11:57:54.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:57:54.280+0000 7f6390e55640 1 librados: shutdown 2024-03-29T11:57:54.306 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:58:01.113 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:58:01.277 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.269+0000 7fa4b5f0e640 10 monclient: get_monmap_and_config 2024-03-29T11:58:01.277 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.269+0000 7fa4b5f0e640 10 monclient: build_initial_monmap 2024-03-29T11:58:01.277 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.269+0000 7fa4b5f0e640 10 monclient: monmap: 2024-03-29T11:58:01.277 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:58:01.277 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:01.277 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:58:01.270198+0000 2024-03-29T11:58:01.277 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:58:01.270198+0000 2024-03-29T11:58:01.277 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:58:01.278 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:01.278 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:58:01.278 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:58:01.278 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:58:01.278 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:01.280 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.270+0000 7fa4b5f0e640 1 Processor -- start 2024-03-29T11:58:01.280 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.270+0000 7fa4b5f0e640 1 -- start start 2024-03-29T11:58:01.280 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.270+0000 7fa4b5f0e640 10 monclient: init 2024-03-29T11:58:01.280 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.271+0000 7fa4b5f0e640 10 monclient: _reopen_session rank -1 2024-03-29T11:58:01.280 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.271+0000 7fa4b5f0e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:58:01.280 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.271+0000 7fa4b5f0e640 10 monclient(hunting): picked mon.noname-a con 0x7fa4b00ae250 addr v1:172.21.15.98:6789/0 2024-03-29T11:58:01.281 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.271+0000 7fa4b5f0e640 10 monclient(hunting): picked mon.noname-b con 0x7fa4b013f970 addr v1:172.21.15.98:6790/0 2024-03-29T11:58:01.281 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.271+0000 7fa4b5f0e640 10 monclient(hunting): picked mon.noname-c con 0x7fa4b0142800 addr v1:172.21.15.121:6789/0 2024-03-29T11:58:01.281 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.271+0000 7fa4b5f0e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4b00afe00 con 0x7fa4b00ae250 2024-03-29T11:58:01.281 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.271+0000 7fa4b5f0e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4b0142c10 con 0x7fa4b0142800 2024-03-29T11:58:01.281 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.271+0000 7fa4b5f0e640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4b00afb20 con 0x7fa4b013f970 2024-03-29T11:58:01.281 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.271+0000 7fa4b5f0e640 10 monclient(hunting): _renew_subs 2024-03-29T11:58:01.281 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.271+0000 7fa4b5f0e640 10 monclient(hunting): authenticate will time out at 1388.356768s 2024-03-29T11:58:01.281 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.272+0000 7fa4aeffd640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fa4b013f970 0x7fa4b01420f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:60826/0 (socket says 172.21.15.98:60826) 2024-03-29T11:58:01.281 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.272+0000 7fa4aeffd640 1 -- 172.21.15.98:0/3415667140 learned_addr learned my addr 172.21.15.98:0/3415667140 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:58:01.281 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.272+0000 7fa4ae7fc640 1 -- 172.21.15.98:0/3415667140 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3866214602 0 0) 0x7fa4b00afb20 con 0x7fa4b013f970 2024-03-29T11:58:01.281 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.272+0000 7fa4ae7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:58:01.281 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.272+0000 7fa4ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:58:01.281 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.272+0000 7fa4ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:58:01.281 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.272+0000 7fa4ae7fc640 10 monclient(hunting): my global_id is 4604 2024-03-29T11:58:01.281 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.272+0000 7fa4ae7fc640 1 -- 172.21.15.98:0/3415667140 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa498003520 con 0x7fa4b013f970 2024-03-29T11:58:01.282 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.273+0000 7fa4ae7fc640 1 -- 172.21.15.98:0/3415667140 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2807073065 0 0) 0x7fa498003520 con 0x7fa4b013f970 2024-03-29T11:58:01.282 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.273+0000 7fa4ae7fc640 1 -- 172.21.15.98:0/3415667140 >> v1:172.21.15.121:6789/0 conn(0x7fa4b0142800 legacy=0x7fa4b014da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:58:01.282 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.273+0000 7fa4ae7fc640 1 -- 172.21.15.98:0/3415667140 >> v1:172.21.15.98:6789/0 conn(0x7fa4b00ae250 legacy=0x7fa4b013f260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:58:01.283 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.274+0000 7fa4ae7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:58:01.283 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.274+0000 7fa4ae7fc640 1 monclient: found mon.noname-b 2024-03-29T11:58:01.283 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.274+0000 7fa4ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:58:01.283 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.274+0000 7fa4ae7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:58:01.283 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.274+0000 7fa4ae7fc640 1 -- 172.21.15.98:0/3415667140 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b00aab80 con 0x7fa4b013f970 2024-03-29T11:58:01.283 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.274+0000 7fa4ae7fc640 10 monclient: _finish_auth 0 2024-03-29T11:58:01.283 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.274+0000 7fa4ae7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:58:01.283 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.274+0000 7fa4ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:58:01.283 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.274+0000 7fa4ae7fc640 1 -- 172.21.15.98:0/3415667140 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fa4a0001670 con 0x7fa4b013f970 2024-03-29T11:58:01.283 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.274+0000 7fa4ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:58:01.283 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.274+0000 7fa4ae7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:58:01.284 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.274+0000 7fa4ae7fc640 10 monclient: dump: 2024-03-29T11:58:01.284 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:58:01.284 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:01.284 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:01.284 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:01.284 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:58:01.284 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:01.284 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:58:01.284 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:58:01.284 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:58:01.284 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:01.284 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.274+0000 7fa4b5f0e640 5 monclient: authenticate success, global_id 4604 2024-03-29T11:58:01.284 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.274+0000 7fa4b5f0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:58:01.284 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.275+0000 7fa4ae7fc640 1 -- 172.21.15.98:0/3415667140 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4a00018d0 con 0x7fa4b013f970 2024-03-29T11:58:01.284 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.275+0000 7fa4ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:58:01.285 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.275+0000 7fa4ae7fc640 1 -- 172.21.15.98:0/3415667140 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fa4a0001d20 con 0x7fa4b013f970 2024-03-29T11:58:01.285 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.275+0000 7fa4ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:58:01.285 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.275+0000 7fa4ae7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:58:01.285 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.275+0000 7fa4ae7fc640 10 monclient: dump: 2024-03-29T11:58:01.285 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:58:01.285 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:01.285 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:01.285 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:01.285 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:58:01.285 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:01.285 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:58:01.285 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:58:01.285 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:58:01.285 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:01.285 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.275+0000 7fa4b5f0e640 10 monclient: get_monmap_and_config success 2024-03-29T11:58:01.286 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.275+0000 7fa4b5f0e640 4 set_mon_vals no callback set 2024-03-29T11:58:01.286 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.276+0000 7fa4b5f0e640 10 monclient: shutdown 2024-03-29T11:58:01.286 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.276+0000 7fa4b5f0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:58:01.286 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.276+0000 7fa4b5f0e640 1 -- 172.21.15.98:0/3415667140 >> v1:172.21.15.98:6790/0 conn(0x7fa4b013f970 legacy=0x7fa4b01420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:01.287 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.276+0000 7fa4b5f0e640 1 -- 172.21.15.98:0/3415667140 shutdown_connections 2024-03-29T11:58:01.287 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.276+0000 7fa4b5f0e640 1 -- 172.21.15.98:0/3415667140 >> 172.21.15.98:0/3415667140 conn(0x7fa4b00b35e0 msgr2=0x7fa4b00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:58:01.287 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.276+0000 7fa4b5f0e640 1 -- 172.21.15.98:0/3415667140 shutdown_connections 2024-03-29T11:58:01.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.276+0000 7fa4b5f0e640 1 -- 172.21.15.98:0/3415667140 wait complete. 2024-03-29T11:58:01.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.277+0000 7fa4b5f0e640 10 monclient: build_initial_monmap 2024-03-29T11:58:01.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.277+0000 7fa4b5f0e640 10 monclient: monmap: 2024-03-29T11:58:01.288 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:58:01.288 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:58:01.288 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:58:01.288 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:58:01.288 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:58:01.288 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:01.288 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:58:01.288 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:58:01.288 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:58:01.288 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:01.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.278+0000 7fa4b5f0e640 1 librados: starting msgr at 2024-03-29T11:58:01.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.278+0000 7fa4b5f0e640 1 librados: starting objecter 2024-03-29T11:58:01.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.278+0000 7fa4b5f0e640 1 Processor -- start 2024-03-29T11:58:01.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.278+0000 7fa4b5f0e640 1 -- start start 2024-03-29T11:58:01.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.278+0000 7fa4b5f0e640 1 librados: setting wanted keys 2024-03-29T11:58:01.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.278+0000 7fa4b5f0e640 1 librados: calling monclient init 2024-03-29T11:58:01.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.278+0000 7fa4b5f0e640 10 monclient: init 2024-03-29T11:58:01.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.279+0000 7fa4b5f0e640 10 monclient: _reopen_session rank -1 2024-03-29T11:58:01.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.279+0000 7fa4b5f0e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:58:01.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.282+0000 7fa4b5f0e640 10 monclient(hunting): picked mon.noname-a con 0x7fa4b00ae250 addr v1:172.21.15.98:6789/0 2024-03-29T11:58:01.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.282+0000 7fa4b5f0e640 10 monclient(hunting): picked mon.noname-b con 0x7fa4b013f970 addr v1:172.21.15.121:6789/0 2024-03-29T11:58:01.292 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.283+0000 7fa4af7fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa4b00ae250 0x7fa4b0071bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:58693/0 (socket says 172.21.15.98:58693) 2024-03-29T11:58:01.292 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.283+0000 7fa4affff640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fa4b0142800 0x7fa4b01e6cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:36734/0 (socket says 172.21.15.98:36734) 2024-03-29T11:58:01.292 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.283+0000 7fa4affff640 1 -- 172.21.15.98:0/1934284952 learned_addr learned my addr 172.21.15.98:0/1934284952 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:58:01.292 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.283+0000 7fa4b5f0e640 10 monclient(hunting): picked mon.noname-c con 0x7fa4b0142800 addr v1:172.21.15.98:6790/0 2024-03-29T11:58:01.292 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.283+0000 7fa497fff640 10 client.?.objecter ms_handle_connect 0x7fa4b0142800 2024-03-29T11:58:01.292 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.283+0000 7fa497fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:58:01.292 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.283+0000 7fa4b5f0e640 1 -- 172.21.15.98:0/1934284952 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4b0138070 con 0x7fa4b00ae250 2024-03-29T11:58:01.292 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.283+0000 7fa4b5f0e640 1 -- 172.21.15.98:0/1934284952 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4b0146350 con 0x7fa4b013f970 2024-03-29T11:58:01.292 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.284+0000 7fa4b5f0e640 1 -- 172.21.15.98:0/1934284952 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4b0144c40 con 0x7fa4b0142800 2024-03-29T11:58:01.292 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.284+0000 7fa4b5f0e640 10 monclient(hunting): _renew_subs 2024-03-29T11:58:01.292 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.284+0000 7fa4b5f0e640 10 monclient(hunting): authenticate will time out at 1388.369332s 2024-03-29T11:58:01.292 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.284+0000 7fa497fff640 10 client.?.objecter ms_handle_connect 0x7fa4b00ae250 2024-03-29T11:58:01.293 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.284+0000 7fa497fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:58:01.293 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.284+0000 7fa497fff640 1 -- 172.21.15.98:0/1934284952 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2626814820 0 0) 0x7fa4b0144c40 con 0x7fa4b0142800 2024-03-29T11:58:01.293 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.284+0000 7fa497fff640 10 monclient(hunting): _negotiate 2024-03-29T11:58:01.293 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.284+0000 7fa497fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:58:01.293 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.285+0000 7fa497fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:58:01.293 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.285+0000 7fa497fff640 10 monclient(hunting): my global_id is 4607 2024-03-29T11:58:01.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.285+0000 7fa497fff640 1 -- 172.21.15.98:0/1934284952 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa49c004eb0 con 0x7fa4b0142800 2024-03-29T11:58:01.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.285+0000 7fa497fff640 1 -- 172.21.15.98:0/1934284952 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 699616620 0 0) 0x7fa4b0138070 con 0x7fa4b00ae250 2024-03-29T11:58:01.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.285+0000 7fa497fff640 10 monclient(hunting): _negotiate 2024-03-29T11:58:01.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.285+0000 7fa497fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:58:01.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.285+0000 7fa497fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:58:01.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.286+0000 7fa497fff640 10 monclient(hunting): my global_id is 4557 2024-03-29T11:58:01.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.286+0000 7fa497fff640 1 -- 172.21.15.98:0/1934284952 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4b0144c40 con 0x7fa4b00ae250 2024-03-29T11:58:01.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.286+0000 7fa497fff640 1 -- 172.21.15.98:0/1934284952 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1530376246 0 0) 0x7fa49c004eb0 con 0x7fa4b0142800 2024-03-29T11:58:01.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.286+0000 7fa497fff640 1 -- 172.21.15.98:0/1934284952 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4b0138070 con 0x7fa4b0142800 2024-03-29T11:58:01.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.286+0000 7fa497fff640 1 -- 172.21.15.98:0/1934284952 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fa4a8001ad0 con 0x7fa4b0142800 2024-03-29T11:58:01.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.286+0000 7fa497fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:58:01.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.286+0000 7fa497fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:58:01.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.286+0000 7fa497fff640 10 monclient(hunting): dump: 2024-03-29T11:58:01.297 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:58:01.298 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:01.298 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:01.298 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:01.298 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:58:01.298 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:01.298 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:58:01.298 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:58:01.298 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:58:01.298 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:01.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.286+0000 7fa497fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:58:01.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.286+0000 7fa497fff640 1 -- 172.21.15.98:0/1934284952 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 164253095 0 0) 0x7fa4b0138070 con 0x7fa4b0142800 2024-03-29T11:58:01.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.286+0000 7fa497fff640 1 -- 172.21.15.98:0/1934284952 >> v1:172.21.15.121:6789/0 conn(0x7fa4b013f970 legacy=0x7fa4b013a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:01.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.286+0000 7fa497fff640 1 -- 172.21.15.98:0/1934284952 >> v1:172.21.15.98:6789/0 conn(0x7fa4b00ae250 legacy=0x7fa4b0071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:01.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.286+0000 7fa497fff640 10 monclient: _finish_hunting 0 2024-03-29T11:58:01.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.286+0000 7fa497fff640 1 monclient: found mon.c 2024-03-29T11:58:01.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.286+0000 7fa497fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:58:01.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.286+0000 7fa497fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:58:01.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.286+0000 7fa497fff640 1 -- 172.21.15.98:0/1934284952 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b0143470 con 0x7fa4b0142800 2024-03-29T11:58:01.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.286+0000 7fa497fff640 10 monclient: _finish_auth 0 2024-03-29T11:58:01.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.286+0000 7fa497fff640 10 monclient: _check_auth_tickets 2024-03-29T11:58:01.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.286+0000 7fa497fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:58:01.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.286+0000 7fa4b5f0e640 5 monclient: authenticate success, global_id 4607 2024-03-29T11:58:01.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.286+0000 7fa4b5f0e640 10 monclient: _renew_subs 2024-03-29T11:58:01.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.286+0000 7fa4b5f0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:58:01.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.286+0000 7fa4b5f0e640 1 -- 172.21.15.98:0/1934284952 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4b0143f60 con 0x7fa4b0142800 2024-03-29T11:58:01.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.286+0000 7fa4b5f0e640 10 client.4607.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:58:01.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.286+0000 7fa4b5f0e640 10 monclient: _renew_subs 2024-03-29T11:58:01.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.286+0000 7fa4b5f0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:58:01.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.286+0000 7fa4b5f0e640 1 -- 172.21.15.98:0/1934284952 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4b0144800 con 0x7fa4b0142800 2024-03-29T11:58:01.300 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.286+0000 7fa4b5f0e640 1 librados: init done 2024-03-29T11:58:01.300 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.288+0000 7fa4b5f0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:58:01.300 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.288+0000 7fa4b5f0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:58:01.300 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.288+0000 7fa4b5f0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:58:01.300 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.288+0000 7fa4b5f0e640 1 -- 172.21.15.98:0/1934284952 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4b00ae630 con 0x7fa4b0142800 2024-03-29T11:58:01.300 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.291+0000 7fa497fff640 1 -- 172.21.15.98:0/1934284952 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4a8001fe0 con 0x7fa4b0142800 2024-03-29T11:58:01.300 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.291+0000 7fa497fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:58:01.301 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.291+0000 7fa497fff640 1 -- 172.21.15.98:0/1934284952 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fa4a8004be0 con 0x7fa4b0142800 2024-03-29T11:58:01.301 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.291+0000 7fa497fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:58:01.301 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.291+0000 7fa497fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:58:01.301 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.291+0000 7fa497fff640 10 monclient: dump: 2024-03-29T11:58:01.301 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:58:01.301 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:01.301 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:01.301 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:01.301 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:58:01.301 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:01.301 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:58:01.301 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:58:01.301 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:58:01.301 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:01.301 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.291+0000 7fa4ad7fa640 4 set_mon_vals no callback set 2024-03-29T11:58:01.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.293+0000 7fa497fff640 1 -- 172.21.15.98:0/1934284952 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7fa4a801abb0 con 0x7fa4b0142800 2024-03-29T11:58:01.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.294+0000 7fa497fff640 1 -- 172.21.15.98:0/1934284952 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7fa4a8084b20 con 0x7fa4b0142800 2024-03-29T11:58:01.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.294+0000 7fa497fff640 10 client.4607.objecter ms_dispatch 0x7fa4b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:58:01.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.294+0000 7fa497fff640 3 client.4607.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:58:01.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.294+0000 7fa497fff640 3 client.4607.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:58:01.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.294+0000 7fa497fff640 20 client.4607.objecter dump_active .. 0 homeless 2024-03-29T11:58:01.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.295+0000 7fa4af7fe640 10 monclient: get_auth_request con 0x7fa49c06dcb0 auth_method 0 2024-03-29T11:58:01.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.298+0000 7fa497fff640 10 client.4607.objecter ms_handle_connect 0x7fa49c06dcb0 2024-03-29T11:58:01.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.298+0000 7fa497fff640 1 -- 172.21.15.98:0/1934284952 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa4a80507e0 con 0x7fa4b0142800 2024-03-29T11:58:01.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.298+0000 7fa497fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:58:01.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.298+0000 7fa497fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:58:01.673 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.666+0000 7fa4b5f0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:58:01.673 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.666+0000 7fa4b5f0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:58:01.674 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.666+0000 7fa4b5f0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:58:01.674 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.666+0000 7fa4b5f0e640 1 -- 172.21.15.98:0/1934284952 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa4b0144300 con 0x7fa4b0142800 2024-03-29T11:58:01.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.670+0000 7fa497fff640 1 -- 172.21.15.98:0/1934284952 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7fa4a8052330 con 0x7fa4b0142800 2024-03-29T11:58:01.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.670+0000 7fa497fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:58:01.678 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.670+0000 7fa497fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:58:01.678 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:58:01.679 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:58:01.683 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.675+0000 7fa4b5f0e640 10 librados: watch_flush enter 2024-03-29T11:58:01.684 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.676+0000 7fa4b5f0e640 10 librados: watch_flush exit 2024-03-29T11:58:01.684 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.676+0000 7fa4b5f0e640 20 client.4607.objecter shutdown clearing up homeless session... 2024-03-29T11:58:01.684 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.676+0000 7fa4b5f0e640 10 client.4607.objecter successfully canceled tick 2024-03-29T11:58:01.685 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.677+0000 7fa4b5f0e640 1 -- 172.21.15.98:0/1934284952 >> v1:172.21.15.98:6816/4065186974 conn(0x7fa49c06dcb0 legacy=0x7fa49c070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:01.685 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.677+0000 7fa4b5f0e640 10 monclient: shutdown 2024-03-29T11:58:01.685 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.677+0000 7fa4b5f0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:58:01.685 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.677+0000 7fa4b5f0e640 1 -- 172.21.15.98:0/1934284952 >> v1:172.21.15.98:6790/0 conn(0x7fa4b0142800 legacy=0x7fa4b01e6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:01.685 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.677+0000 7fa4b5f0e640 1 -- 172.21.15.98:0/1934284952 shutdown_connections 2024-03-29T11:58:01.685 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.677+0000 7fa4b5f0e640 1 -- 172.21.15.98:0/1934284952 >> 172.21.15.98:0/1934284952 conn(0x7fa4b00b35e0 msgr2=0x7fa4b00b2cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:58:01.685 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.677+0000 7fa4b5f0e640 1 -- 172.21.15.98:0/1934284952 shutdown_connections 2024-03-29T11:58:01.685 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.677+0000 7fa4b5f0e640 1 -- 172.21.15.98:0/1934284952 wait complete. 2024-03-29T11:58:01.685 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:01.678+0000 7fa4b5f0e640 1 librados: shutdown 2024-03-29T11:58:01.702 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:58:08.508 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:58:08.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.646+0000 7fe5774a8640 10 monclient: get_monmap_and_config 2024-03-29T11:58:08.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.646+0000 7fe5774a8640 10 monclient: build_initial_monmap 2024-03-29T11:58:08.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.646+0000 7fe5774a8640 10 monclient: monmap: 2024-03-29T11:58:08.654 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:58:08.654 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:08.654 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:58:08.647210+0000 2024-03-29T11:58:08.654 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:58:08.647210+0000 2024-03-29T11:58:08.654 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:58:08.654 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:08.654 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:58:08.654 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:58:08.655 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:58:08.655 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:08.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.647+0000 7fe5774a8640 1 Processor -- start 2024-03-29T11:58:08.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.647+0000 7fe5774a8640 1 -- start start 2024-03-29T11:58:08.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.647+0000 7fe5774a8640 10 monclient: init 2024-03-29T11:58:08.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.648+0000 7fe5774a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:58:08.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.648+0000 7fe5774a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:58:08.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.648+0000 7fe5774a8640 10 monclient(hunting): picked mon.noname-a con 0x7fe5700b0860 addr v1:172.21.15.98:6789/0 2024-03-29T11:58:08.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.648+0000 7fe5774a8640 10 monclient(hunting): picked mon.noname-b con 0x7fe57013ef30 addr v1:172.21.15.98:6790/0 2024-03-29T11:58:08.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.648+0000 7fe5774a8640 10 monclient(hunting): picked mon.noname-c con 0x7fe5700af8e0 addr v1:172.21.15.121:6789/0 2024-03-29T11:58:08.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.648+0000 7fe5774a8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5700b0c40 con 0x7fe5700b0860 2024-03-29T11:58:08.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.648+0000 7fe5774a8640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5700aed10 con 0x7fe5700af8e0 2024-03-29T11:58:08.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.648+0000 7fe5774a8640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5700afd50 con 0x7fe57013ef30 2024-03-29T11:58:08.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.648+0000 7fe5774a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:58:08.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.648+0000 7fe5774a8640 10 monclient(hunting): authenticate will time out at 1395.733766s 2024-03-29T11:58:08.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.648+0000 7fe575a45640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fe5700af8e0 0x7fe570149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:33684/0 (socket says 172.21.15.98:33684) 2024-03-29T11:58:08.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.648+0000 7fe575a45640 1 -- 172.21.15.98:0/1716081067 learned_addr learned my addr 172.21.15.98:0/1716081067 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:58:08.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.649+0000 7fe567fff640 1 -- 172.21.15.98:0/1716081067 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1078958397 0 0) 0x7fe5700b0c40 con 0x7fe5700b0860 2024-03-29T11:58:08.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.649+0000 7fe567fff640 10 monclient(hunting): _negotiate 2024-03-29T11:58:08.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.649+0000 7fe567fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:58:08.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.649+0000 7fe567fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:58:08.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.649+0000 7fe567fff640 10 monclient(hunting): my global_id is 4560 2024-03-29T11:58:08.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.649+0000 7fe567fff640 1 -- 172.21.15.98:0/1716081067 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe55c004630 con 0x7fe5700b0860 2024-03-29T11:58:08.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.649+0000 7fe567fff640 1 -- 172.21.15.98:0/1716081067 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 946343827 0 0) 0x7fe5700aed10 con 0x7fe5700af8e0 2024-03-29T11:58:08.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.649+0000 7fe567fff640 10 monclient(hunting): _negotiate 2024-03-29T11:58:08.660 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.649+0000 7fe567fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:58:08.660 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.649+0000 7fe567fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:58:08.660 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.649+0000 7fe567fff640 10 monclient(hunting): my global_id is 4558 2024-03-29T11:58:08.660 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.649+0000 7fe567fff640 1 -- 172.21.15.98:0/1716081067 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5700b0c40 con 0x7fe5700af8e0 2024-03-29T11:58:08.660 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.649+0000 7fe567fff640 1 -- 172.21.15.98:0/1716081067 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2640485216 0 0) 0x7fe55c004630 con 0x7fe5700b0860 2024-03-29T11:58:08.660 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.649+0000 7fe567fff640 1 -- 172.21.15.98:0/1716081067 >> v1:172.21.15.98:6790/0 conn(0x7fe57013ef30 legacy=0x7fe5700af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:08.660 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.649+0000 7fe567fff640 1 -- 172.21.15.98:0/1716081067 >> v1:172.21.15.121:6789/0 conn(0x7fe5700af8e0 legacy=0x7fe570149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:08.660 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.650+0000 7fe567fff640 10 monclient: _finish_hunting 0 2024-03-29T11:58:08.660 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.650+0000 7fe567fff640 1 monclient: found mon.noname-a 2024-03-29T11:58:08.660 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.650+0000 7fe567fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:58:08.660 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.650+0000 7fe567fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T11:58:08.660 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.650+0000 7fe567fff640 1 -- 172.21.15.98:0/1716081067 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5700ab1b0 con 0x7fe5700b0860 2024-03-29T11:58:08.660 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.650+0000 7fe567fff640 10 monclient: _finish_auth 0 2024-03-29T11:58:08.660 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.650+0000 7fe567fff640 10 monclient: _check_auth_tickets 2024-03-29T11:58:08.660 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.650+0000 7fe567fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:58:08.661 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.650+0000 7fe567fff640 1 -- 172.21.15.98:0/1716081067 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fe56c001690 con 0x7fe5700b0860 2024-03-29T11:58:08.661 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.650+0000 7fe5774a8640 5 monclient: authenticate success, global_id 4560 2024-03-29T11:58:08.661 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.650+0000 7fe5774a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:58:08.661 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.650+0000 7fe567fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:58:08.661 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.650+0000 7fe567fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:58:08.661 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.650+0000 7fe567fff640 10 monclient: dump: 2024-03-29T11:58:08.661 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:58:08.661 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:08.661 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:08.661 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:08.661 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:58:08.661 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:08.661 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:58:08.661 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:58:08.662 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:58:08.662 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:08.662 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.650+0000 7fe5774a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:58:08.662 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.650+0000 7fe567fff640 1 -- 172.21.15.98:0/1716081067 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe56c0017f0 con 0x7fe5700b0860 2024-03-29T11:58:08.662 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.650+0000 7fe567fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:58:08.662 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.650+0000 7fe5774a8640 10 monclient: get_monmap_and_config success 2024-03-29T11:58:08.662 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.650+0000 7fe5774a8640 4 set_mon_vals no callback set 2024-03-29T11:58:08.662 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.651+0000 7fe567fff640 1 -- 172.21.15.98:0/1716081067 <== mon.0 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fe56c001c00 con 0x7fe5700b0860 2024-03-29T11:58:08.662 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.651+0000 7fe567fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:58:08.662 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.651+0000 7fe567fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:58:08.662 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.651+0000 7fe567fff640 10 monclient: dump: 2024-03-29T11:58:08.662 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:58:08.662 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:08.663 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:08.663 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:08.663 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:58:08.663 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:08.663 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:58:08.663 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:58:08.663 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:58:08.663 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:08.672 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.651+0000 7fe5774a8640 10 monclient: shutdown 2024-03-29T11:58:08.672 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.651+0000 7fe5774a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:58:08.672 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.651+0000 7fe5774a8640 1 -- 172.21.15.98:0/1716081067 >> v1:172.21.15.98:6789/0 conn(0x7fe5700b0860 legacy=0x7fe57013e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:08.672 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.651+0000 7fe5774a8640 1 -- 172.21.15.98:0/1716081067 shutdown_connections 2024-03-29T11:58:08.672 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.651+0000 7fe5774a8640 1 -- 172.21.15.98:0/1716081067 >> 172.21.15.98:0/1716081067 conn(0x7fe57006b660 msgr2=0x7fe57006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:58:08.672 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.652+0000 7fe5774a8640 1 -- 172.21.15.98:0/1716081067 shutdown_connections 2024-03-29T11:58:08.672 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.652+0000 7fe5774a8640 1 -- 172.21.15.98:0/1716081067 wait complete. 2024-03-29T11:58:08.672 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.652+0000 7fe5774a8640 10 monclient: build_initial_monmap 2024-03-29T11:58:08.673 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.652+0000 7fe5774a8640 10 monclient: monmap: 2024-03-29T11:58:08.673 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:58:08.673 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:58:08.673 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:58:08.673 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:58:08.673 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:58:08.673 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:08.673 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:58:08.673 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:58:08.673 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:58:08.673 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:08.673 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.653+0000 7fe5774a8640 1 librados: starting msgr at 2024-03-29T11:58:08.673 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.653+0000 7fe5774a8640 1 librados: starting objecter 2024-03-29T11:58:08.673 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.653+0000 7fe5774a8640 1 Processor -- start 2024-03-29T11:58:08.674 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.653+0000 7fe5774a8640 1 -- start start 2024-03-29T11:58:08.674 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.653+0000 7fe5774a8640 1 librados: setting wanted keys 2024-03-29T11:58:08.674 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.653+0000 7fe5774a8640 1 librados: calling monclient init 2024-03-29T11:58:08.674 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.653+0000 7fe5774a8640 10 monclient: init 2024-03-29T11:58:08.674 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.654+0000 7fe5774a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:58:08.674 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.654+0000 7fe5774a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:58:08.674 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.654+0000 7fe5774a8640 10 monclient(hunting): picked mon.noname-a con 0x7fe5700af8e0 addr v1:172.21.15.98:6789/0 2024-03-29T11:58:08.674 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.654+0000 7fe5774a8640 10 monclient(hunting): picked mon.noname-c con 0x7fe5700b0860 addr v1:172.21.15.98:6790/0 2024-03-29T11:58:08.674 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.654+0000 7fe5774a8640 10 monclient(hunting): picked mon.noname-b con 0x7fe57013ef30 addr v1:172.21.15.121:6789/0 2024-03-29T11:58:08.674 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.654+0000 7fe5774a8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5701c8f40 con 0x7fe5700af8e0 2024-03-29T11:58:08.674 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.654+0000 7fe5774a8640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5701c6750 con 0x7fe57013ef30 2024-03-29T11:58:08.674 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.654+0000 7fe5774a8640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5701c7040 con 0x7fe5700b0860 2024-03-29T11:58:08.674 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.654+0000 7fe5774a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:58:08.675 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.654+0000 7fe5774a8640 10 monclient(hunting): authenticate will time out at 1395.740023s 2024-03-29T11:58:08.675 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.655+0000 7fe575244640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe5700af8e0 0x7fe570139380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56056/0 (socket says 172.21.15.98:56056) 2024-03-29T11:58:08.675 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.655+0000 7fe575244640 1 -- 172.21.15.98:0/3839112535 learned_addr learned my addr 172.21.15.98:0/3839112535 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:58:08.675 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.655+0000 7fe565ffb640 10 client.?.objecter ms_handle_connect 0x7fe5700af8e0 2024-03-29T11:58:08.675 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.655+0000 7fe565ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:58:08.675 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.655+0000 7fe565ffb640 10 client.?.objecter ms_handle_connect 0x7fe5700b0860 2024-03-29T11:58:08.675 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.655+0000 7fe565ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:58:08.675 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.655+0000 7fe565ffb640 10 client.?.objecter ms_handle_connect 0x7fe57013ef30 2024-03-29T11:58:08.675 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.656+0000 7fe565ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:58:08.675 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.656+0000 7fe565ffb640 1 -- 172.21.15.98:0/3839112535 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3232004860 0 0) 0x7fe5701c7040 con 0x7fe5700b0860 2024-03-29T11:58:08.675 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.656+0000 7fe565ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:58:08.675 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.656+0000 7fe565ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:58:08.675 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.656+0000 7fe565ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:58:08.675 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.656+0000 7fe565ffb640 10 monclient(hunting): my global_id is 4613 2024-03-29T11:58:08.675 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.656+0000 7fe565ffb640 1 -- 172.21.15.98:0/3839112535 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe554004eb0 con 0x7fe5700b0860 2024-03-29T11:58:08.676 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.656+0000 7fe565ffb640 1 -- 172.21.15.98:0/3839112535 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1588676343 0 0) 0x7fe5701c8f40 con 0x7fe5700af8e0 2024-03-29T11:58:08.676 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.656+0000 7fe565ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:58:08.676 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.656+0000 7fe565ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:58:08.676 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.656+0000 7fe565ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:58:08.676 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.656+0000 7fe565ffb640 10 monclient(hunting): my global_id is 4563 2024-03-29T11:58:08.676 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.656+0000 7fe565ffb640 1 -- 172.21.15.98:0/3839112535 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5701c7040 con 0x7fe5700af8e0 2024-03-29T11:58:08.676 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.656+0000 7fe565ffb640 1 -- 172.21.15.98:0/3839112535 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1373780818 0 0) 0x7fe554004eb0 con 0x7fe5700b0860 2024-03-29T11:58:08.676 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.656+0000 7fe565ffb640 1 -- 172.21.15.98:0/3839112535 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5701c8f40 con 0x7fe5700b0860 2024-03-29T11:58:08.676 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.656+0000 7fe565ffb640 1 -- 172.21.15.98:0/3839112535 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fe560001810 con 0x7fe5700b0860 2024-03-29T11:58:08.676 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.657+0000 7fe565ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:58:08.676 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.657+0000 7fe565ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:58:08.676 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.657+0000 7fe565ffb640 10 monclient(hunting): dump: 2024-03-29T11:58:08.676 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:58:08.676 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:08.677 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:08.677 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:08.677 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:58:08.677 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:08.677 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:58:08.677 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:58:08.677 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:58:08.677 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:08.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.657+0000 7fe565ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:58:08.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.657+0000 7fe565ffb640 1 -- 172.21.15.98:0/3839112535 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 555040970 0 0) 0x7fe5701c6750 con 0x7fe57013ef30 2024-03-29T11:58:08.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.657+0000 7fe565ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:58:08.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.657+0000 7fe565ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:58:08.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.657+0000 7fe565ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:58:08.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.657+0000 7fe565ffb640 10 monclient(hunting): my global_id is 4561 2024-03-29T11:58:08.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.657+0000 7fe565ffb640 1 -- 172.21.15.98:0/3839112535 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe554004eb0 con 0x7fe57013ef30 2024-03-29T11:58:08.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.657+0000 7fe565ffb640 1 -- 172.21.15.98:0/3839112535 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 777155755 0 0) 0x7fe5701c8f40 con 0x7fe5700b0860 2024-03-29T11:58:08.678 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.657+0000 7fe565ffb640 1 -- 172.21.15.98:0/3839112535 >> v1:172.21.15.121:6789/0 conn(0x7fe57013ef30 legacy=0x7fe5701e6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:08.678 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.657+0000 7fe565ffb640 1 -- 172.21.15.98:0/3839112535 >> v1:172.21.15.98:6789/0 conn(0x7fe5700af8e0 legacy=0x7fe570139380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:08.678 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.657+0000 7fe565ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:58:08.678 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.657+0000 7fe565ffb640 1 monclient: found mon.c 2024-03-29T11:58:08.678 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.657+0000 7fe565ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:58:08.678 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.657+0000 7fe565ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:58:08.678 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.657+0000 7fe565ffb640 1 -- 172.21.15.98:0/3839112535 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5701c7320 con 0x7fe5700b0860 2024-03-29T11:58:08.678 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.657+0000 7fe565ffb640 10 monclient: _finish_auth 0 2024-03-29T11:58:08.678 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.657+0000 7fe565ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:58:08.678 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.657+0000 7fe565ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:58:08.678 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.657+0000 7fe5774a8640 5 monclient: authenticate success, global_id 4613 2024-03-29T11:58:08.678 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.657+0000 7fe5774a8640 10 monclient: _renew_subs 2024-03-29T11:58:08.678 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.657+0000 7fe5774a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:58:08.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.657+0000 7fe5774a8640 1 -- 172.21.15.98:0/3839112535 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5701c7600 con 0x7fe5700b0860 2024-03-29T11:58:08.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.657+0000 7fe5774a8640 10 client.4613.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:58:08.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.657+0000 7fe5774a8640 10 monclient: _renew_subs 2024-03-29T11:58:08.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.657+0000 7fe5774a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:58:08.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.657+0000 7fe5774a8640 1 -- 172.21.15.98:0/3839112535 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5701df480 con 0x7fe5700b0860 2024-03-29T11:58:08.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.657+0000 7fe5774a8640 1 librados: init done 2024-03-29T11:58:08.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.659+0000 7fe565ffb640 1 -- 172.21.15.98:0/3839112535 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe560001d00 con 0x7fe5700b0860 2024-03-29T11:58:08.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.659+0000 7fe565ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:58:08.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.659+0000 7fe5774a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:58:08.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.659+0000 7fe5677fe640 4 set_mon_vals no callback set 2024-03-29T11:58:08.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.659+0000 7fe565ffb640 1 -- 172.21.15.98:0/3839112535 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fe5600037b0 con 0x7fe5700b0860 2024-03-29T11:58:08.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.659+0000 7fe565ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:58:08.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.659+0000 7fe565ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:58:08.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.659+0000 7fe565ffb640 10 monclient: dump: 2024-03-29T11:58:08.680 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:58:08.680 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:08.680 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:08.680 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:08.680 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:58:08.680 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:08.680 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:58:08.680 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:58:08.680 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:58:08.680 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:08.680 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.660+0000 7fe5774a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:58:08.680 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.660+0000 7fe5774a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:58:08.680 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.660+0000 7fe5774a8640 1 -- 172.21.15.98:0/3839112535 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5701c8d10 con 0x7fe5700b0860 2024-03-29T11:58:08.680 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.660+0000 7fe565ffb640 1 -- 172.21.15.98:0/3839112535 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7fe5600039d0 con 0x7fe5700b0860 2024-03-29T11:58:08.680 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.661+0000 7fe575244640 10 monclient: get_auth_request con 0x7fe55406de00 auth_method 0 2024-03-29T11:58:08.681 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.661+0000 7fe565ffb640 1 -- 172.21.15.98:0/3839112535 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7fe560083760 con 0x7fe5700b0860 2024-03-29T11:58:08.681 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.661+0000 7fe565ffb640 10 client.4613.objecter ms_dispatch 0x7fe570000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:58:08.681 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.661+0000 7fe565ffb640 3 client.4613.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:58:08.681 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.661+0000 7fe565ffb640 3 client.4613.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:58:08.681 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.661+0000 7fe565ffb640 20 client.4613.objecter dump_active .. 0 homeless 2024-03-29T11:58:08.681 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.661+0000 7fe565ffb640 10 client.4613.objecter ms_handle_connect 0x7fe55406de00 2024-03-29T11:58:08.681 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.668+0000 7fe565ffb640 1 -- 172.21.15.98:0/3839112535 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe5701c8d10 con 0x7fe5700b0860 2024-03-29T11:58:08.682 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.669+0000 7fe565ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:58:08.682 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:08.669+0000 7fe565ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:58:09.031 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:09.024+0000 7fe5774a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:58:09.031 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:09.024+0000 7fe5774a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:58:09.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:09.024+0000 7fe5774a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:58:09.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:09.024+0000 7fe5774a8640 1 -- 172.21.15.98:0/3839112535 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5701c8d10 con 0x7fe5700b0860 2024-03-29T11:58:09.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:09.025+0000 7fe565ffb640 1 -- 172.21.15.98:0/3839112535 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7fe5701c8d10 con 0x7fe5700b0860 2024-03-29T11:58:09.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:09.025+0000 7fe565ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:58:09.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:09.025+0000 7fe565ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:58:09.033 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:58:09.033 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:58:09.036 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:09.028+0000 7fe5774a8640 10 librados: watch_flush enter 2024-03-29T11:58:09.036 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:09.028+0000 7fe5774a8640 10 librados: watch_flush exit 2024-03-29T11:58:09.036 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:09.028+0000 7fe5774a8640 20 client.4613.objecter shutdown clearing up homeless session... 2024-03-29T11:58:09.036 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:09.028+0000 7fe5774a8640 10 client.4613.objecter successfully canceled tick 2024-03-29T11:58:09.036 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:09.029+0000 7fe5774a8640 1 -- 172.21.15.98:0/3839112535 >> v1:172.21.15.98:6816/4065186974 conn(0x7fe55406de00 legacy=0x7fe5540702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:09.036 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:09.029+0000 7fe5774a8640 10 monclient: shutdown 2024-03-29T11:58:09.036 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:09.029+0000 7fe5774a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:58:09.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:09.029+0000 7fe5774a8640 1 -- 172.21.15.98:0/3839112535 >> v1:172.21.15.98:6790/0 conn(0x7fe5700b0860 legacy=0x7fe5701dfb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:09.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:09.029+0000 7fe5774a8640 1 -- 172.21.15.98:0/3839112535 shutdown_connections 2024-03-29T11:58:09.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:09.029+0000 7fe5774a8640 1 -- 172.21.15.98:0/3839112535 >> 172.21.15.98:0/3839112535 conn(0x7fe57006b660 msgr2=0x7fe57013dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:58:09.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:09.029+0000 7fe5774a8640 1 -- 172.21.15.98:0/3839112535 shutdown_connections 2024-03-29T11:58:09.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:09.030+0000 7fe5774a8640 1 -- 172.21.15.98:0/3839112535 wait complete. 2024-03-29T11:58:09.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:09.030+0000 7fe5774a8640 1 librados: shutdown 2024-03-29T11:58:09.052 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:58:15.860 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:58:16.006 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:15.998+0000 7f7dfcb54640 10 monclient: get_monmap_and_config 2024-03-29T11:58:16.006 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:15.998+0000 7f7dfcb54640 10 monclient: build_initial_monmap 2024-03-29T11:58:16.006 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:15.998+0000 7f7dfcb54640 10 monclient: monmap: 2024-03-29T11:58:16.006 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:58:16.006 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:16.006 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:58:15.999859+0000 2024-03-29T11:58:16.006 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:58:15.999859+0000 2024-03-29T11:58:16.006 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:58:16.007 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:16.007 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:58:16.007 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:58:16.007 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:58:16.007 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:16.008 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:15.999+0000 7f7dfcb54640 1 Processor -- start 2024-03-29T11:58:16.008 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:15.999+0000 7f7dfcb54640 1 -- start start 2024-03-29T11:58:16.008 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:15.999+0000 7f7dfcb54640 10 monclient: init 2024-03-29T11:58:16.008 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.000+0000 7f7dfcb54640 10 monclient: _reopen_session rank -1 2024-03-29T11:58:16.008 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.000+0000 7f7dfcb54640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:58:16.008 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.000+0000 7f7dfcb54640 10 monclient(hunting): picked mon.noname-c con 0x7f7df81422b0 addr v1:172.21.15.121:6789/0 2024-03-29T11:58:16.008 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.000+0000 7f7dfcb54640 10 monclient(hunting): picked mon.noname-a con 0x7f7df8142690 addr v1:172.21.15.98:6789/0 2024-03-29T11:58:16.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.000+0000 7f7dfcb54640 10 monclient(hunting): picked mon.noname-b con 0x7f7df80b17d0 addr v1:172.21.15.98:6790/0 2024-03-29T11:58:16.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.000+0000 7f7dfcb54640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7df8142c80 con 0x7f7df8142690 2024-03-29T11:58:16.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.001+0000 7f7dfcb54640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7df80aae40 con 0x7f7df81422b0 2024-03-29T11:58:16.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.001+0000 7f7dfcb54640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7df813fbd0 con 0x7f7df80b17d0 2024-03-29T11:58:16.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.001+0000 7f7dfcb54640 10 monclient(hunting): _renew_subs 2024-03-29T11:58:16.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.001+0000 7f7dfcb54640 10 monclient(hunting): authenticate will time out at 1403.086179s 2024-03-29T11:58:16.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.001+0000 7f7df5d9b640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f7df8142690 0x7f7df80b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44512/0 (socket says 172.21.15.98:44512) 2024-03-29T11:58:16.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.001+0000 7f7df5d9b640 1 -- 172.21.15.98:0/3505318388 learned_addr learned my addr 172.21.15.98:0/3505318388 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:58:16.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.001+0000 7f7df559a640 1 -- 172.21.15.98:0/3505318388 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1360881300 0 0) 0x7f7df8142c80 con 0x7f7df8142690 2024-03-29T11:58:16.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.001+0000 7f7df559a640 10 monclient(hunting): _negotiate 2024-03-29T11:58:16.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.001+0000 7f7df559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:58:16.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.001+0000 7f7df559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:58:16.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.001+0000 7f7df559a640 10 monclient(hunting): my global_id is 4566 2024-03-29T11:58:16.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.001+0000 7f7df559a640 1 -- 172.21.15.98:0/3505318388 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7de0003520 con 0x7f7df8142690 2024-03-29T11:58:16.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.002+0000 7f7df559a640 1 -- 172.21.15.98:0/3505318388 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1377551859 0 0) 0x7f7de0003520 con 0x7f7df8142690 2024-03-29T11:58:16.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.002+0000 7f7df559a640 1 -- 172.21.15.98:0/3505318388 >> v1:172.21.15.98:6790/0 conn(0x7f7df80b17d0 legacy=0x7f7df80b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:58:16.012 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.004+0000 7f7df559a640 1 -- 172.21.15.98:0/3505318388 >> v1:172.21.15.121:6789/0 conn(0x7f7df81422b0 legacy=0x7f7df813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:16.012 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.004+0000 7f7df559a640 10 monclient: _finish_hunting 0 2024-03-29T11:58:16.012 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.004+0000 7f7df559a640 1 monclient: found mon.noname-a 2024-03-29T11:58:16.012 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.004+0000 7f7df559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:58:16.012 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.004+0000 7f7df559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T11:58:16.012 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.004+0000 7f7df559a640 1 -- 172.21.15.98:0/3505318388 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df8151940 con 0x7f7df8142690 2024-03-29T11:58:16.012 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.004+0000 7f7df559a640 10 monclient: _finish_auth 0 2024-03-29T11:58:16.012 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.004+0000 7f7df559a640 10 monclient: _check_auth_tickets 2024-03-29T11:58:16.012 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.004+0000 7f7df559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:58:16.012 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.004+0000 7f7df559a640 1 -- 172.21.15.98:0/3505318388 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f7dec001690 con 0x7f7df8142690 2024-03-29T11:58:16.012 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.004+0000 7f7df559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:58:16.012 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.004+0000 7f7df559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:58:16.012 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.004+0000 7f7df559a640 10 monclient: dump: 2024-03-29T11:58:16.012 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:58:16.013 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:16.013 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:16.013 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:16.013 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:58:16.013 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:16.013 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:58:16.013 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:58:16.013 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:58:16.013 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:16.013 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.004+0000 7f7dfcb54640 5 monclient: authenticate success, global_id 4566 2024-03-29T11:58:16.013 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.004+0000 7f7dfcb54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:58:16.015 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.004+0000 7f7df559a640 1 -- 172.21.15.98:0/3505318388 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7dec0018b0 con 0x7f7df8142690 2024-03-29T11:58:16.015 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.004+0000 7f7df559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:58:16.015 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.004+0000 7f7df559a640 1 -- 172.21.15.98:0/3505318388 <== mon.0 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f7dec001c40 con 0x7f7df8142690 2024-03-29T11:58:16.015 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.004+0000 7f7df559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:58:16.015 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.004+0000 7f7df559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:58:16.015 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.004+0000 7f7df559a640 10 monclient: dump: 2024-03-29T11:58:16.015 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:58:16.015 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:16.015 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:16.015 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:16.016 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:58:16.016 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:16.016 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:58:16.016 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:58:16.016 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:58:16.016 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:16.016 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.004+0000 7f7dfcb54640 10 monclient: get_monmap_and_config success 2024-03-29T11:58:16.016 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.004+0000 7f7dfcb54640 4 set_mon_vals no callback set 2024-03-29T11:58:16.016 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.005+0000 7f7dfcb54640 10 monclient: shutdown 2024-03-29T11:58:16.016 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.005+0000 7f7dfcb54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:58:16.016 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.005+0000 7f7dfcb54640 1 -- 172.21.15.98:0/3505318388 >> v1:172.21.15.98:6789/0 conn(0x7f7df8142690 legacy=0x7f7df80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:16.016 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.006+0000 7f7dfcb54640 1 -- 172.21.15.98:0/3505318388 shutdown_connections 2024-03-29T11:58:16.016 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.006+0000 7f7dfcb54640 1 -- 172.21.15.98:0/3505318388 >> 172.21.15.98:0/3505318388 conn(0x7f7df8066a00 msgr2=0x7f7df8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:58:16.016 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.006+0000 7f7dfcb54640 1 -- 172.21.15.98:0/3505318388 shutdown_connections 2024-03-29T11:58:16.017 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.006+0000 7f7dfcb54640 1 -- 172.21.15.98:0/3505318388 wait complete. 2024-03-29T11:58:16.017 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.007+0000 7f7dfcb54640 10 monclient: build_initial_monmap 2024-03-29T11:58:16.017 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.007+0000 7f7dfcb54640 10 monclient: monmap: 2024-03-29T11:58:16.017 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:58:16.017 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:58:16.017 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:58:16.017 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:58:16.017 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:58:16.017 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:16.017 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:58:16.017 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:58:16.017 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:58:16.017 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:16.017 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.007+0000 7f7dfcb54640 1 librados: starting msgr at 2024-03-29T11:58:16.017 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.007+0000 7f7dfcb54640 1 librados: starting objecter 2024-03-29T11:58:16.017 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.007+0000 7f7dfcb54640 1 Processor -- start 2024-03-29T11:58:16.018 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.007+0000 7f7dfcb54640 1 -- start start 2024-03-29T11:58:16.018 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.007+0000 7f7dfcb54640 1 librados: setting wanted keys 2024-03-29T11:58:16.018 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.007+0000 7f7dfcb54640 1 librados: calling monclient init 2024-03-29T11:58:16.018 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.007+0000 7f7dfcb54640 10 monclient: init 2024-03-29T11:58:16.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.008+0000 7f7dfcb54640 10 monclient: _reopen_session rank -1 2024-03-29T11:58:16.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.009+0000 7f7dfcb54640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:58:16.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.009+0000 7f7dfcb54640 10 monclient(hunting): picked mon.noname-b con 0x7f7df80b17d0 addr v1:172.21.15.121:6789/0 2024-03-29T11:58:16.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.009+0000 7f7dfcb54640 10 monclient(hunting): picked mon.noname-c con 0x7f7df81422b0 addr v1:172.21.15.98:6790/0 2024-03-29T11:58:16.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.009+0000 7f7df659c640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f7df80b17d0 0x7f7df8150c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:51398/0 (socket says 172.21.15.98:51398) 2024-03-29T11:58:16.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.009+0000 7f7df6d9d640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f7df8142690 0x7f7df81e1c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44884/0 (socket says 172.21.15.98:44884) 2024-03-29T11:58:16.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.009+0000 7f7dfcb54640 10 monclient(hunting): picked mon.noname-a con 0x7f7df8142690 addr v1:172.21.15.98:6789/0 2024-03-29T11:58:16.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.009+0000 7f7dfcb54640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7df80ace80 con 0x7f7df8142690 2024-03-29T11:58:16.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.009+0000 7f7df659c640 1 -- 172.21.15.98:0/786187283 learned_addr learned my addr 172.21.15.98:0/786187283 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:58:16.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.009+0000 7f7dfcb54640 1 -- 172.21.15.98:0/786187283 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7df813eea0 con 0x7f7df80b17d0 2024-03-29T11:58:16.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.009+0000 7f7dfcb54640 1 -- 172.21.15.98:0/786187283 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7df814f9a0 con 0x7f7df81422b0 2024-03-29T11:58:16.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.009+0000 7f7dfcb54640 10 monclient(hunting): _renew_subs 2024-03-29T11:58:16.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.009+0000 7f7dfcb54640 10 monclient(hunting): authenticate will time out at 1403.094913s 2024-03-29T11:58:16.021 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.009+0000 7f7de6ffd640 10 client.?.objecter ms_handle_connect 0x7f7df8142690 2024-03-29T11:58:16.021 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.009+0000 7f7de6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:58:16.021 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.010+0000 7f7de6ffd640 1 -- 172.21.15.98:0/786187283 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 520473319 0 0) 0x7f7df80ace80 con 0x7f7df8142690 2024-03-29T11:58:16.021 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.010+0000 7f7de6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:58:16.021 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.010+0000 7f7de6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:58:16.021 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.010+0000 7f7de6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:58:16.021 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.010+0000 7f7de6ffd640 10 monclient(hunting): my global_id is 4569 2024-03-29T11:58:16.021 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.010+0000 7f7de6ffd640 1 -- 172.21.15.98:0/786187283 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dd8004eb0 con 0x7f7df8142690 2024-03-29T11:58:16.021 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.010+0000 7f7de6ffd640 1 -- 172.21.15.98:0/786187283 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4038829589 0 0) 0x7f7dd8004eb0 con 0x7f7df8142690 2024-03-29T11:58:16.021 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.010+0000 7f7de6ffd640 1 -- 172.21.15.98:0/786187283 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7df80ace80 con 0x7f7df8142690 2024-03-29T11:58:16.021 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.010+0000 7f7de6ffd640 1 -- 172.21.15.98:0/786187283 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f7df0001ba0 con 0x7f7df8142690 2024-03-29T11:58:16.021 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.010+0000 7f7de6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:58:16.021 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.010+0000 7f7de6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:58:16.021 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.010+0000 7f7de6ffd640 10 monclient(hunting): dump: 2024-03-29T11:58:16.021 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:58:16.022 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:16.022 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:16.022 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:16.022 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:58:16.022 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:16.022 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:58:16.022 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:58:16.022 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:58:16.022 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:16.022 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.010+0000 7f7de6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:58:16.022 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.010+0000 7f7de6ffd640 1 -- 172.21.15.98:0/786187283 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2491599919 0 0) 0x7f7df80ace80 con 0x7f7df8142690 2024-03-29T11:58:16.022 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.010+0000 7f7de6ffd640 1 -- 172.21.15.98:0/786187283 >> v1:172.21.15.98:6790/0 conn(0x7f7df81422b0 legacy=0x7f7df814f290 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:58:16.022 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.012+0000 7f7de6ffd640 1 -- 172.21.15.98:0/786187283 >> v1:172.21.15.121:6789/0 conn(0x7f7df80b17d0 legacy=0x7f7df8150c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:16.022 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.012+0000 7f7de6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:58:16.022 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.012+0000 7f7de6ffd640 1 monclient: found mon.a 2024-03-29T11:58:16.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.012+0000 7f7de6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:58:16.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.012+0000 7f7de6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:58:16.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.012+0000 7f7de6ffd640 1 -- 172.21.15.98:0/786187283 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df80abac0 con 0x7f7df8142690 2024-03-29T11:58:16.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.012+0000 7f7de6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:58:16.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.012+0000 7f7de6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:58:16.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.012+0000 7f7de6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:58:16.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.012+0000 7f7de6ffd640 10 client.?.objecter ms_handle_connect 0x7f7df80b17d0 2024-03-29T11:58:16.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.012+0000 7f7de6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:58:16.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.012+0000 7f7dfcb54640 5 monclient: authenticate success, global_id 4569 2024-03-29T11:58:16.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.012+0000 7f7dfcb54640 10 monclient: _renew_subs 2024-03-29T11:58:16.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.012+0000 7f7dfcb54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:58:16.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.012+0000 7f7dfcb54640 1 -- 172.21.15.98:0/786187283 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7df80abd10 con 0x7f7df8142690 2024-03-29T11:58:16.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.012+0000 7f7dfcb54640 10 client.4569.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:58:16.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.012+0000 7f7dfcb54640 10 monclient: _renew_subs 2024-03-29T11:58:16.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.012+0000 7f7dfcb54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:58:16.024 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.012+0000 7f7dfcb54640 1 -- 172.21.15.98:0/786187283 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7df813e4f0 con 0x7f7df8142690 2024-03-29T11:58:16.024 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.012+0000 7f7dfcb54640 1 librados: init done 2024-03-29T11:58:16.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.013+0000 7f7dfcb54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:58:16.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.013+0000 7f7dfcb54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:58:16.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.013+0000 7f7dfcb54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:58:16.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.013+0000 7f7dfcb54640 1 -- 172.21.15.98:0/786187283 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7df812db50 con 0x7f7df8142690 2024-03-29T11:58:16.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.014+0000 7f7de6ffd640 1 -- 172.21.15.98:0/786187283 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7df0001d00 con 0x7f7df8142690 2024-03-29T11:58:16.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.014+0000 7f7de6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:58:16.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.014+0000 7f7de6ffd640 1 -- 172.21.15.98:0/786187283 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f7df0003b90 con 0x7f7df8142690 2024-03-29T11:58:16.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.014+0000 7f7de6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:58:16.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.014+0000 7f7de6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:58:16.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.014+0000 7f7de6ffd640 10 monclient: dump: 2024-03-29T11:58:16.025 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:58:16.026 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:16.026 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:16.026 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:16.026 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:58:16.026 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:16.026 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:58:16.026 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:58:16.026 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:58:16.026 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:16.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.014+0000 7f7df4d99640 4 set_mon_vals no callback set 2024-03-29T11:58:16.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.015+0000 7f7de6ffd640 1 -- 172.21.15.98:0/786187283 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f7df0003db0 con 0x7f7df8142690 2024-03-29T11:58:16.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.015+0000 7f7de6ffd640 1 -- 172.21.15.98:0/786187283 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 452813045 0 0) 0x7f7df0084d80 con 0x7f7df8142690 2024-03-29T11:58:16.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.015+0000 7f7de6ffd640 10 client.4569.objecter ms_dispatch 0x7f7df8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:58:16.027 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.015+0000 7f7de6ffd640 3 client.4569.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:58:16.027 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.015+0000 7f7de6ffd640 3 client.4569.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:58:16.027 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.015+0000 7f7de6ffd640 20 client.4569.objecter dump_active .. 0 homeless 2024-03-29T11:58:16.027 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.016+0000 7f7df659c640 10 monclient: get_auth_request con 0x7f7dd806dbc0 auth_method 0 2024-03-29T11:58:16.027 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.016+0000 7f7de6ffd640 10 client.4569.objecter ms_handle_connect 0x7f7dd806dbc0 2024-03-29T11:58:16.029 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.022+0000 7f7de6ffd640 1 -- 172.21.15.98:0/786187283 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7df812db50 con 0x7f7df8142690 2024-03-29T11:58:16.029 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.022+0000 7f7de6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:58:16.029 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.022+0000 7f7de6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:58:16.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.335+0000 7f7dfcb54640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:58:16.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.335+0000 7f7dfcb54640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:58:16.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.335+0000 7f7dfcb54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:58:16.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.335+0000 7f7dfcb54640 1 -- 172.21.15.98:0/786187283 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7df812db50 con 0x7f7df8142690 2024-03-29T11:58:16.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.336+0000 7f7de6ffd640 1 -- 172.21.15.98:0/786187283 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 821729074) 0x7f7df812db50 con 0x7f7df8142690 2024-03-29T11:58:16.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.336+0000 7f7de6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:58:16.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.336+0000 7f7de6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:58:16.344 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:58:16.344 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:48:20.065203+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:48:20.065203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:58:16.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.339+0000 7f7dfcb54640 10 librados: watch_flush enter 2024-03-29T11:58:16.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.339+0000 7f7dfcb54640 10 librados: watch_flush exit 2024-03-29T11:58:16.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.339+0000 7f7dfcb54640 20 client.4569.objecter shutdown clearing up homeless session... 2024-03-29T11:58:16.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.339+0000 7f7dfcb54640 10 client.4569.objecter successfully canceled tick 2024-03-29T11:58:16.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.339+0000 7f7dfcb54640 1 -- 172.21.15.98:0/786187283 >> v1:172.21.15.98:6816/4065186974 conn(0x7f7dd806dbc0 legacy=0x7f7dd8070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:16.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.339+0000 7f7dfcb54640 10 monclient: shutdown 2024-03-29T11:58:16.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.339+0000 7f7dfcb54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:58:16.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.339+0000 7f7dfcb54640 1 -- 172.21.15.98:0/786187283 >> v1:172.21.15.98:6789/0 conn(0x7f7df8142690 legacy=0x7f7df81e1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:16.348 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.339+0000 7f7dfcb54640 1 -- 172.21.15.98:0/786187283 shutdown_connections 2024-03-29T11:58:16.348 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.339+0000 7f7dfcb54640 1 -- 172.21.15.98:0/786187283 >> 172.21.15.98:0/786187283 conn(0x7f7df8066a00 msgr2=0x7f7df813c230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:58:16.348 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.339+0000 7f7dfcb54640 1 -- 172.21.15.98:0/786187283 shutdown_connections 2024-03-29T11:58:16.348 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.340+0000 7f7dfcb54640 1 -- 172.21.15.98:0/786187283 wait complete. 2024-03-29T11:58:16.348 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:16.340+0000 7f7dfcb54640 1 librados: shutdown 2024-03-29T11:58:16.360 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:58:23.168 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:58:23.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.308+0000 7f34a83af640 10 monclient: get_monmap_and_config 2024-03-29T11:58:23.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.308+0000 7f34a83af640 10 monclient: build_initial_monmap 2024-03-29T11:58:23.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.308+0000 7f34a83af640 10 monclient: monmap: 2024-03-29T11:58:23.317 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:58:23.317 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:23.317 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:58:23.310300+0000 2024-03-29T11:58:23.317 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:58:23.310300+0000 2024-03-29T11:58:23.317 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:58:23.317 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:23.317 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:58:23.317 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:58:23.317 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:58:23.317 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:23.318 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.309+0000 7f34a83af640 1 Processor -- start 2024-03-29T11:58:23.318 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.309+0000 7f34a83af640 1 -- start start 2024-03-29T11:58:23.318 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.309+0000 7f34a83af640 10 monclient: init 2024-03-29T11:58:23.318 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.310+0000 7f34a83af640 10 monclient: _reopen_session rank -1 2024-03-29T11:58:23.319 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.311+0000 7f34a83af640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:58:23.319 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.311+0000 7f34a614b640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f34a00b0860 0x7f34a013e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55092/0 (socket says 172.21.15.98:55092) 2024-03-29T11:58:23.319 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.311+0000 7f34a614b640 1 -- 172.21.15.98:0/2456758948 learned_addr learned my addr 172.21.15.98:0/2456758948 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:58:23.319 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.311+0000 7f34a83af640 10 monclient(hunting): picked mon.noname-a con 0x7f34a00b0860 addr v1:172.21.15.98:6789/0 2024-03-29T11:58:23.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.312+0000 7f34a83af640 10 monclient(hunting): picked mon.noname-b con 0x7f34a006d9d0 addr v1:172.21.15.98:6790/0 2024-03-29T11:58:23.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.312+0000 7f34a83af640 10 monclient(hunting): picked mon.noname-c con 0x7f34a013ef30 addr v1:172.21.15.121:6789/0 2024-03-29T11:58:23.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.312+0000 7f34a83af640 1 -- 172.21.15.98:0/2456758948 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34a00b0c40 con 0x7f34a00b0860 2024-03-29T11:58:23.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.312+0000 7f34a83af640 1 -- 172.21.15.98:0/2456758948 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34a00aed10 con 0x7f34a013ef30 2024-03-29T11:58:23.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.312+0000 7f34a83af640 1 -- 172.21.15.98:0/2456758948 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34a00ab1b0 con 0x7f34a006d9d0 2024-03-29T11:58:23.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.312+0000 7f34a83af640 10 monclient(hunting): _renew_subs 2024-03-29T11:58:23.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.312+0000 7f34a83af640 10 monclient(hunting): authenticate will time out at 1410.399036s 2024-03-29T11:58:23.323 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.314+0000 7f34a5149640 1 -- 172.21.15.98:0/2456758948 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 287020133 0 0) 0x7f34a00b0c40 con 0x7f34a00b0860 2024-03-29T11:58:23.323 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.315+0000 7f34a5149640 10 monclient(hunting): _negotiate 2024-03-29T11:58:23.323 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.315+0000 7f34a5149640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:58:23.323 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.315+0000 7f34a5149640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:58:23.323 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.315+0000 7f34a5149640 10 monclient(hunting): my global_id is 4572 2024-03-29T11:58:23.323 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.315+0000 7f34a5149640 1 -- 172.21.15.98:0/2456758948 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f348c003520 con 0x7f34a00b0860 2024-03-29T11:58:23.329 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.315+0000 7f34a5149640 1 -- 172.21.15.98:0/2456758948 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3485269686 0 0) 0x7f348c003520 con 0x7f34a00b0860 2024-03-29T11:58:23.329 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.315+0000 7f34a5149640 1 -- 172.21.15.98:0/2456758948 >> v1:172.21.15.98:6790/0 conn(0x7f34a006d9d0 legacy=0x7f34a00af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:23.329 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.315+0000 7f34a5149640 1 -- 172.21.15.98:0/2456758948 >> v1:172.21.15.121:6789/0 conn(0x7f34a013ef30 legacy=0x7f34a0149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:23.329 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.315+0000 7f34a5149640 10 monclient: _finish_hunting 0 2024-03-29T11:58:23.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.315+0000 7f34a5149640 1 monclient: found mon.noname-a 2024-03-29T11:58:23.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.315+0000 7f34a5149640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:58:23.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.315+0000 7f34a5149640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T11:58:23.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.315+0000 7f34a5149640 1 -- 172.21.15.98:0/2456758948 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a00ab490 con 0x7f34a00b0860 2024-03-29T11:58:23.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.316+0000 7f34a5149640 10 monclient: _finish_auth 0 2024-03-29T11:58:23.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.316+0000 7f34a5149640 10 monclient: _check_auth_tickets 2024-03-29T11:58:23.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.316+0000 7f34a5149640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:58:23.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.316+0000 7f34a5149640 1 -- 172.21.15.98:0/2456758948 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f349c001690 con 0x7f34a00b0860 2024-03-29T11:58:23.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.316+0000 7f34a5149640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:58:23.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.316+0000 7f34a5149640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:58:23.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.316+0000 7f34a5149640 10 monclient: dump: 2024-03-29T11:58:23.330 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:58:23.330 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:23.331 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:23.331 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:23.331 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:58:23.331 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:23.331 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:58:23.331 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:58:23.331 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:58:23.331 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:23.331 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.316+0000 7f34a5149640 1 -- 172.21.15.98:0/2456758948 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f349c0018b0 con 0x7f34a00b0860 2024-03-29T11:58:23.331 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.316+0000 7f34a5149640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:58:23.331 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.316+0000 7f34a5149640 1 -- 172.21.15.98:0/2456758948 <== mon.0 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f349c001c40 con 0x7f34a00b0860 2024-03-29T11:58:23.331 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.316+0000 7f34a5149640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:58:23.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.316+0000 7f34a5149640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:58:23.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.316+0000 7f34a5149640 10 monclient: dump: 2024-03-29T11:58:23.332 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:58:23.332 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:23.332 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:23.332 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:23.332 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:58:23.332 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:23.332 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:58:23.332 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:58:23.332 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:58:23.332 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:23.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.316+0000 7f34a83af640 5 monclient: authenticate success, global_id 4572 2024-03-29T11:58:23.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.316+0000 7f34a83af640 10 monclient: get_monmap_and_config success 2024-03-29T11:58:23.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.316+0000 7f34a83af640 4 set_mon_vals no callback set 2024-03-29T11:58:23.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.317+0000 7f34a83af640 10 monclient: shutdown 2024-03-29T11:58:23.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.317+0000 7f34a83af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:58:23.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.317+0000 7f34a83af640 1 -- 172.21.15.98:0/2456758948 >> v1:172.21.15.98:6789/0 conn(0x7f34a00b0860 legacy=0x7f34a013e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:23.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.317+0000 7f34a83af640 1 -- 172.21.15.98:0/2456758948 shutdown_connections 2024-03-29T11:58:23.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.317+0000 7f34a83af640 1 -- 172.21.15.98:0/2456758948 >> 172.21.15.98:0/2456758948 conn(0x7f34a006b660 msgr2=0x7f34a006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:58:23.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.318+0000 7f34a83af640 1 -- 172.21.15.98:0/2456758948 shutdown_connections 2024-03-29T11:58:23.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.318+0000 7f34a83af640 1 -- 172.21.15.98:0/2456758948 wait complete. 2024-03-29T11:58:23.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.319+0000 7f34a83af640 10 monclient: build_initial_monmap 2024-03-29T11:58:23.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.320+0000 7f34a83af640 10 monclient: monmap: 2024-03-29T11:58:23.333 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:58:23.333 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:58:23.334 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:58:23.334 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:58:23.334 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:58:23.334 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:23.334 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:58:23.334 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:58:23.334 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:58:23.334 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:23.334 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.320+0000 7f34a83af640 1 librados: starting msgr at 2024-03-29T11:58:23.334 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.320+0000 7f34a83af640 1 librados: starting objecter 2024-03-29T11:58:23.334 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.320+0000 7f34a83af640 1 Processor -- start 2024-03-29T11:58:23.334 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.320+0000 7f34a83af640 1 -- start start 2024-03-29T11:58:23.334 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.320+0000 7f34a83af640 1 librados: setting wanted keys 2024-03-29T11:58:23.335 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.321+0000 7f34a83af640 1 librados: calling monclient init 2024-03-29T11:58:23.335 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.321+0000 7f34a83af640 10 monclient: init 2024-03-29T11:58:23.340 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.322+0000 7f34a83af640 10 monclient: _reopen_session rank -1 2024-03-29T11:58:23.340 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.322+0000 7f34a83af640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:58:23.340 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.322+0000 7f34a83af640 10 monclient(hunting): picked mon.noname-c con 0x7f34a006d9d0 addr v1:172.21.15.98:6790/0 2024-03-29T11:58:23.340 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.322+0000 7f34a83af640 10 monclient(hunting): picked mon.noname-a con 0x7f34a00b0860 addr v1:172.21.15.98:6789/0 2024-03-29T11:58:23.340 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.322+0000 7f34a83af640 10 monclient(hunting): picked mon.noname-b con 0x7f34a013ef30 addr v1:172.21.15.121:6789/0 2024-03-29T11:58:23.340 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.322+0000 7f34a83af640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34a0139e80 con 0x7f34a00b0860 2024-03-29T11:58:23.340 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.322+0000 7f34a83af640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34a0135710 con 0x7f34a013ef30 2024-03-29T11:58:23.340 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.322+0000 7f34a83af640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34a0136800 con 0x7f34a006d9d0 2024-03-29T11:58:23.340 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.322+0000 7f34a83af640 10 monclient(hunting): _renew_subs 2024-03-29T11:58:23.340 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.322+0000 7f34a83af640 10 monclient(hunting): authenticate will time out at 1410.408779s 2024-03-29T11:58:23.340 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.323+0000 7f34a594a640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f34a00b0860 0x7f34a0132780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55370/0 (socket says 172.21.15.98:55370) 2024-03-29T11:58:23.340 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.323+0000 7f34a594a640 1 -- 172.21.15.98:0/4134848672 learned_addr learned my addr 172.21.15.98:0/4134848672 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:58:23.341 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.323+0000 7f3496ffd640 10 client.?.objecter ms_handle_connect 0x7f34a00b0860 2024-03-29T11:58:23.341 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.324+0000 7f3496ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:58:23.341 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.324+0000 7f3496ffd640 1 -- 172.21.15.98:0/4134848672 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2016374141 0 0) 0x7f34a0139e80 con 0x7f34a00b0860 2024-03-29T11:58:23.341 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.324+0000 7f3496ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:58:23.341 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.324+0000 7f3496ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:58:23.341 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.324+0000 7f3496ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:58:23.341 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.324+0000 7f3496ffd640 10 monclient(hunting): my global_id is 4575 2024-03-29T11:58:23.341 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.324+0000 7f3496ffd640 1 -- 172.21.15.98:0/4134848672 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3484004eb0 con 0x7f34a00b0860 2024-03-29T11:58:23.341 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.324+0000 7f3496ffd640 1 -- 172.21.15.98:0/4134848672 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3473618604 0 0) 0x7f3484004eb0 con 0x7f34a00b0860 2024-03-29T11:58:23.341 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.324+0000 7f3496ffd640 1 -- 172.21.15.98:0/4134848672 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3484005300 con 0x7f34a00b0860 2024-03-29T11:58:23.341 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.324+0000 7f3496ffd640 1 -- 172.21.15.98:0/4134848672 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f34900019f0 con 0x7f34a00b0860 2024-03-29T11:58:23.341 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.324+0000 7f3496ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:58:23.341 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.324+0000 7f3496ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:58:23.341 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.324+0000 7f3496ffd640 10 monclient(hunting): dump: 2024-03-29T11:58:23.342 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:58:23.342 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:23.342 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:23.342 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:23.342 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:58:23.342 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:23.342 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:58:23.342 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:58:23.342 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:58:23.342 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:23.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.324+0000 7f3496ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:58:23.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.325+0000 7f3496ffd640 1 -- 172.21.15.98:0/4134848672 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4231383423 0 0) 0x7f3484005300 con 0x7f34a00b0860 2024-03-29T11:58:23.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.325+0000 7f3496ffd640 1 -- 172.21.15.98:0/4134848672 >> v1:172.21.15.98:6790/0 conn(0x7f34a006d9d0 legacy=0x7f34a01360f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:58:23.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.325+0000 7f3496ffd640 1 -- 172.21.15.98:0/4134848672 >> v1:172.21.15.121:6789/0 conn(0x7f34a013ef30 legacy=0x7f34a01d35e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:58:23.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.325+0000 7f3496ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:58:23.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.326+0000 7f3496ffd640 1 monclient: found mon.a 2024-03-29T11:58:23.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.326+0000 7f3496ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:58:23.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.326+0000 7f3496ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:58:23.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.326+0000 7f3496ffd640 1 -- 172.21.15.98:0/4134848672 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a0136990 con 0x7f34a00b0860 2024-03-29T11:58:23.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.326+0000 7f3496ffd640 10 monclient: _finish_auth 0 2024-03-29T11:58:23.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.326+0000 7f3496ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:58:23.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.326+0000 7f3496ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:58:23.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.326+0000 7f34a83af640 5 monclient: authenticate success, global_id 4575 2024-03-29T11:58:23.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.326+0000 7f34a83af640 10 monclient: _renew_subs 2024-03-29T11:58:23.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.326+0000 7f34a83af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:58:23.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.326+0000 7f34a83af640 1 -- 172.21.15.98:0/4134848672 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34a0138120 con 0x7f34a00b0860 2024-03-29T11:58:23.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.326+0000 7f34a83af640 10 client.4575.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:58:23.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.326+0000 7f34a83af640 10 monclient: _renew_subs 2024-03-29T11:58:23.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.326+0000 7f34a83af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:58:23.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.326+0000 7f34a83af640 1 -- 172.21.15.98:0/4134848672 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f34a0132e90 con 0x7f34a00b0860 2024-03-29T11:58:23.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.326+0000 7f34a83af640 1 librados: init done 2024-03-29T11:58:23.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.327+0000 7f34a83af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:58:23.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.327+0000 7f34a83af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:58:23.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.327+0000 7f34a83af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:58:23.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.327+0000 7f34a83af640 1 -- 172.21.15.98:0/4134848672 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34a01d3220 con 0x7f34a00b0860 2024-03-29T11:58:23.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.328+0000 7f3496ffd640 1 -- 172.21.15.98:0/4134848672 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3490001cf0 con 0x7f34a00b0860 2024-03-29T11:58:23.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.328+0000 7f3496ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:58:23.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.328+0000 7f3496ffd640 1 -- 172.21.15.98:0/4134848672 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f34900019f0 con 0x7f34a00b0860 2024-03-29T11:58:23.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.328+0000 7f3496ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:58:23.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.328+0000 7f3496ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:58:23.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.328+0000 7f3496ffd640 10 monclient: dump: 2024-03-29T11:58:23.344 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:58:23.344 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:23.344 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:23.345 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:23.345 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:58:23.345 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:23.345 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:58:23.345 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:58:23.345 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:58:23.345 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:23.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.328+0000 7f3497fff640 4 set_mon_vals no callback set 2024-03-29T11:58:23.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.330+0000 7f3496ffd640 1 -- 172.21.15.98:0/4134848672 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f34900039c0 con 0x7f34a00b0860 2024-03-29T11:58:23.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.330+0000 7f3496ffd640 1 -- 172.21.15.98:0/4134848672 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3033626086 0 0) 0x7f3490084890 con 0x7f34a00b0860 2024-03-29T11:58:23.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.330+0000 7f3496ffd640 10 client.4575.objecter ms_dispatch 0x7f34a0000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:58:23.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.330+0000 7f3496ffd640 3 client.4575.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:58:23.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.330+0000 7f3496ffd640 3 client.4575.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:58:23.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.330+0000 7f3496ffd640 20 client.4575.objecter dump_active .. 0 homeless 2024-03-29T11:58:23.346 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.331+0000 7f34a614b640 10 monclient: get_auth_request con 0x7f348406e120 auth_method 0 2024-03-29T11:58:23.346 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.331+0000 7f3496ffd640 10 client.4575.objecter ms_handle_connect 0x7f348406e120 2024-03-29T11:58:23.346 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.337+0000 7f3496ffd640 1 -- 172.21.15.98:0/4134848672 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3490050170 con 0x7f34a00b0860 2024-03-29T11:58:23.346 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.337+0000 7f3496ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:58:23.346 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.337+0000 7f3496ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:58:23.670 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.662+0000 7f34a83af640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:58:23.670 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.662+0000 7f34a83af640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:58:23.671 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.662+0000 7f34a83af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:58:23.671 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.662+0000 7f34a83af640 1 -- 172.21.15.98:0/4134848672 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34a0075f10 con 0x7f34a00b0860 2024-03-29T11:58:23.672 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.664+0000 7f3496ffd640 1 -- 172.21.15.98:0/4134848672 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3141891755) 0x7f34900520a0 con 0x7f34a00b0860 2024-03-29T11:58:23.672 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.664+0000 7f3496ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:58:23.672 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.664+0000 7f3496ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:58:23.673 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:58:23.673 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":11,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:58:22.774801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:58:22.774801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:58:23.676 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.668+0000 7f34a83af640 10 librados: watch_flush enter 2024-03-29T11:58:23.676 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.668+0000 7f34a83af640 10 librados: watch_flush exit 2024-03-29T11:58:23.676 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.668+0000 7f34a83af640 20 client.4575.objecter shutdown clearing up homeless session... 2024-03-29T11:58:23.676 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.668+0000 7f34a83af640 10 client.4575.objecter successfully canceled tick 2024-03-29T11:58:23.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.668+0000 7f34a83af640 1 -- 172.21.15.98:0/4134848672 >> v1:172.21.15.98:6816/4065186974 conn(0x7f348406e120 legacy=0x7f34840705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:23.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.668+0000 7f34a83af640 10 monclient: shutdown 2024-03-29T11:58:23.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.668+0000 7f34a83af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:58:23.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.668+0000 7f34a83af640 1 -- 172.21.15.98:0/4134848672 >> v1:172.21.15.98:6789/0 conn(0x7f34a00b0860 legacy=0x7f34a0132780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:23.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.668+0000 7f34a83af640 1 -- 172.21.15.98:0/4134848672 shutdown_connections 2024-03-29T11:58:23.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.668+0000 7f34a83af640 1 -- 172.21.15.98:0/4134848672 >> 172.21.15.98:0/4134848672 conn(0x7f34a006b660 msgr2=0x7f34a0134ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:58:23.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.669+0000 7f34a83af640 1 -- 172.21.15.98:0/4134848672 shutdown_connections 2024-03-29T11:58:23.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.669+0000 7f34a83af640 1 -- 172.21.15.98:0/4134848672 wait complete. 2024-03-29T11:58:23.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:23.669+0000 7f34a83af640 1 librados: shutdown 2024-03-29T11:58:23.696 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:58:30.502 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:58:30.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.644+0000 7f1f0f8a8640 10 monclient: get_monmap_and_config 2024-03-29T11:58:30.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.644+0000 7f1f0f8a8640 10 monclient: build_initial_monmap 2024-03-29T11:58:30.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.644+0000 7f1f0f8a8640 10 monclient: monmap: 2024-03-29T11:58:30.652 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:58:30.652 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:30.652 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:58:30.645736+0000 2024-03-29T11:58:30.652 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:58:30.645736+0000 2024-03-29T11:58:30.652 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:58:30.652 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:30.652 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:58:30.652 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:58:30.653 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:58:30.653 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:30.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.649+0000 7f1f0f8a8640 1 Processor -- start 2024-03-29T11:58:30.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.649+0000 7f1f0f8a8640 1 -- start start 2024-03-29T11:58:30.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.649+0000 7f1f0f8a8640 10 monclient: init 2024-03-29T11:58:30.661 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.650+0000 7f1f0f8a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:58:30.661 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.650+0000 7f1f0f8a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:58:30.661 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.650+0000 7f1f0f8a8640 10 monclient(hunting): picked mon.noname-a con 0x7f1f0806ade0 addr v1:172.21.15.98:6789/0 2024-03-29T11:58:30.661 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.650+0000 7f1f0f8a8640 10 monclient(hunting): picked mon.noname-c con 0x7f1f0813d830 addr v1:172.21.15.121:6789/0 2024-03-29T11:58:30.661 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.650+0000 7f1f0f8a8640 10 monclient(hunting): picked mon.noname-b con 0x7f1f081405a0 addr v1:172.21.15.98:6790/0 2024-03-29T11:58:30.661 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.650+0000 7f1f0f8a8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f08072290 con 0x7f1f0806ade0 2024-03-29T11:58:30.661 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.650+0000 7f1f0f8a8640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f08071f60 con 0x7f1f0813d830 2024-03-29T11:58:30.662 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.650+0000 7f1f0f8a8640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f080b0e00 con 0x7f1f081405a0 2024-03-29T11:58:30.662 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.650+0000 7f1f0f8a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:58:30.662 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.650+0000 7f1f0f8a8640 10 monclient(hunting): authenticate will time out at 1417.735953s 2024-03-29T11:58:30.662 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.654+0000 7f1f0de45640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f1f081405a0 0x7f1f0814b710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:49197/0 (socket says 172.21.15.98:49197) 2024-03-29T11:58:30.662 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.654+0000 7f1f0de45640 1 -- 172.21.15.98:0/27803895 learned_addr learned my addr 172.21.15.98:0/27803895 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:58:30.664 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.655+0000 7f1efffff640 1 -- 172.21.15.98:0/27803895 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3455130360 0 0) 0x7f1f080b0e00 con 0x7f1f081405a0 2024-03-29T11:58:30.664 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.655+0000 7f1efffff640 10 monclient(hunting): _negotiate 2024-03-29T11:58:30.664 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.655+0000 7f1efffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:58:30.664 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.655+0000 7f1efffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:58:30.664 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.655+0000 7f1efffff640 10 monclient(hunting): my global_id is 4619 2024-03-29T11:58:30.664 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.655+0000 7f1efffff640 1 -- 172.21.15.98:0/27803895 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ef4003520 con 0x7f1f081405a0 2024-03-29T11:58:30.664 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.655+0000 7f1efffff640 1 -- 172.21.15.98:0/27803895 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1490815546 0 0) 0x7f1f08072290 con 0x7f1f0806ade0 2024-03-29T11:58:30.664 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.655+0000 7f1efffff640 10 monclient(hunting): _negotiate 2024-03-29T11:58:30.664 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.655+0000 7f1efffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:58:30.664 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.655+0000 7f1efffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:58:30.665 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.655+0000 7f1efffff640 10 monclient(hunting): my global_id is 4578 2024-03-29T11:58:30.665 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.655+0000 7f1efffff640 1 -- 172.21.15.98:0/27803895 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ef4003840 con 0x7f1f0806ade0 2024-03-29T11:58:30.665 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.655+0000 7f1efffff640 1 -- 172.21.15.98:0/27803895 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 418605773 0 0) 0x7f1ef4003520 con 0x7f1f081405a0 2024-03-29T11:58:30.665 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.655+0000 7f1efffff640 1 -- 172.21.15.98:0/27803895 >> v1:172.21.15.121:6789/0 conn(0x7f1f0813d830 legacy=0x7f1f0813fe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:58:30.665 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.656+0000 7f1efffff640 1 -- 172.21.15.98:0/27803895 >> v1:172.21.15.98:6789/0 conn(0x7f1f0806ade0 legacy=0x7f1f0813d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:30.665 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.656+0000 7f1efffff640 10 monclient: _finish_hunting 0 2024-03-29T11:58:30.665 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.656+0000 7f1efffff640 1 monclient: found mon.noname-b 2024-03-29T11:58:30.665 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.656+0000 7f1efffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:58:30.665 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.656+0000 7f1efffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:58:30.665 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.656+0000 7f1efffff640 1 -- 172.21.15.98:0/27803895 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f08070ae0 con 0x7f1f081405a0 2024-03-29T11:58:30.665 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.656+0000 7f1efffff640 10 monclient: _finish_auth 0 2024-03-29T11:58:30.665 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.656+0000 7f1efffff640 10 monclient: _check_auth_tickets 2024-03-29T11:58:30.665 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.656+0000 7f1efffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:58:30.665 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.656+0000 7f1efffff640 1 -- 172.21.15.98:0/27803895 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f1f000019f0 con 0x7f1f081405a0 2024-03-29T11:58:30.666 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.656+0000 7f1efffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:58:30.666 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.656+0000 7f1efffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:58:30.666 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.656+0000 7f1efffff640 10 monclient: dump: 2024-03-29T11:58:30.666 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:58:30.666 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:30.666 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:30.666 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:30.666 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:58:30.666 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:30.666 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:58:30.666 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:58:30.666 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:58:30.666 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:30.666 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.657+0000 7f1f0f8a8640 5 monclient: authenticate success, global_id 4619 2024-03-29T11:58:30.666 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.657+0000 7f1f0f8a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:58:30.667 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.659+0000 7f1efffff640 1 -- 172.21.15.98:0/27803895 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f00001bd0 con 0x7f1f081405a0 2024-03-29T11:58:30.667 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.659+0000 7f1efffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:58:30.668 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.659+0000 7f1efffff640 1 -- 172.21.15.98:0/27803895 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f1f00001fe0 con 0x7f1f081405a0 2024-03-29T11:58:30.668 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.659+0000 7f1efffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:58:30.668 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.659+0000 7f1efffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:58:30.668 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.659+0000 7f1efffff640 10 monclient: dump: 2024-03-29T11:58:30.668 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:58:30.668 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:30.668 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:30.668 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:30.668 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:58:30.668 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:30.668 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:58:30.668 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:58:30.668 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:58:30.668 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:30.669 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.662+0000 7f1f0f8a8640 10 monclient: get_monmap_and_config success 2024-03-29T11:58:30.669 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.662+0000 7f1f0f8a8640 4 set_mon_vals no callback set 2024-03-29T11:58:30.669 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.662+0000 7f1f0f8a8640 10 monclient: shutdown 2024-03-29T11:58:30.670 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.662+0000 7f1f0f8a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:58:30.670 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.662+0000 7f1f0f8a8640 1 -- 172.21.15.98:0/27803895 >> v1:172.21.15.98:6790/0 conn(0x7f1f081405a0 legacy=0x7f1f0814b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:30.670 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.663+0000 7f1f0f8a8640 1 -- 172.21.15.98:0/27803895 shutdown_connections 2024-03-29T11:58:30.670 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.663+0000 7f1f0f8a8640 1 -- 172.21.15.98:0/27803895 >> 172.21.15.98:0/27803895 conn(0x7f1f080b4930 msgr2=0x7f1f080b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:58:30.670 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.663+0000 7f1f0f8a8640 1 -- 172.21.15.98:0/27803895 shutdown_connections 2024-03-29T11:58:30.670 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.663+0000 7f1f0f8a8640 1 -- 172.21.15.98:0/27803895 wait complete. 2024-03-29T11:58:30.671 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.664+0000 7f1f0f8a8640 10 monclient: build_initial_monmap 2024-03-29T11:58:30.671 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.664+0000 7f1f0f8a8640 10 monclient: monmap: 2024-03-29T11:58:30.671 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:58:30.671 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:58:30.671 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:58:30.671 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:58:30.672 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:58:30.672 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:30.672 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:58:30.672 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:58:30.672 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:58:30.672 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:30.672 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.665+0000 7f1f0f8a8640 1 librados: starting msgr at 2024-03-29T11:58:30.672 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.665+0000 7f1f0f8a8640 1 librados: starting objecter 2024-03-29T11:58:30.672 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.665+0000 7f1f0f8a8640 1 Processor -- start 2024-03-29T11:58:30.672 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.665+0000 7f1f0f8a8640 1 -- start start 2024-03-29T11:58:30.673 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.665+0000 7f1f0f8a8640 1 librados: setting wanted keys 2024-03-29T11:58:30.673 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.665+0000 7f1f0f8a8640 1 librados: calling monclient init 2024-03-29T11:58:30.673 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.665+0000 7f1f0f8a8640 10 monclient: init 2024-03-29T11:58:30.673 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.666+0000 7f1f0f8a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:58:30.673 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.666+0000 7f1f0f8a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:58:30.673 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.666+0000 7f1f0f8a8640 10 monclient(hunting): picked mon.noname-b con 0x7f1f0806ade0 addr v1:172.21.15.121:6789/0 2024-03-29T11:58:30.673 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.666+0000 7f1f0f8a8640 10 monclient(hunting): picked mon.noname-a con 0x7f1f0813d830 addr v1:172.21.15.98:6789/0 2024-03-29T11:58:30.673 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.666+0000 7f1f0f8a8640 10 monclient(hunting): picked mon.noname-c con 0x7f1f081405a0 addr v1:172.21.15.98:6790/0 2024-03-29T11:58:30.673 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.666+0000 7f1f0f8a8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f0812f3b0 con 0x7f1f0813d830 2024-03-29T11:58:30.673 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.666+0000 7f1f0f8a8640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f080ac090 con 0x7f1f0806ade0 2024-03-29T11:58:30.673 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.666+0000 7f1f0f8a8640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f080ac370 con 0x7f1f081405a0 2024-03-29T11:58:30.674 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.666+0000 7f1f0f8a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:58:30.674 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.666+0000 7f1f0f8a8640 10 monclient(hunting): authenticate will time out at 1417.751584s 2024-03-29T11:58:30.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.670+0000 7f1f0de45640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f1f081405a0 0x7f1f081e1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:49337/0 (socket says 172.21.15.98:49337) 2024-03-29T11:58:30.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.670+0000 7f1f0de45640 1 -- 172.21.15.98:0/1038734959 learned_addr learned my addr 172.21.15.98:0/1038734959 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:58:30.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.670+0000 7f1efdffb640 10 client.?.objecter ms_handle_connect 0x7f1f081405a0 2024-03-29T11:58:30.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.670+0000 7f1efdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:58:30.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.670+0000 7f1efdffb640 10 client.?.objecter ms_handle_connect 0x7f1f0813d830 2024-03-29T11:58:30.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.670+0000 7f1efdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:58:30.685 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.671+0000 7f1efdffb640 10 client.?.objecter ms_handle_connect 0x7f1f0806ade0 2024-03-29T11:58:30.685 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.671+0000 7f1efdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:58:30.685 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.671+0000 7f1efdffb640 1 -- 172.21.15.98:0/1038734959 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1090737958 0 0) 0x7f1f0812f3b0 con 0x7f1f0813d830 2024-03-29T11:58:30.685 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.671+0000 7f1efdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:58:30.685 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.671+0000 7f1efdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:58:30.685 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.671+0000 7f1efdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:58:30.685 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.671+0000 7f1efdffb640 10 monclient(hunting): my global_id is 4581 2024-03-29T11:58:30.685 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.671+0000 7f1efdffb640 1 -- 172.21.15.98:0/1038734959 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1eec004eb0 con 0x7f1f0813d830 2024-03-29T11:58:30.685 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.671+0000 7f1efdffb640 1 -- 172.21.15.98:0/1038734959 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 203938831 0 0) 0x7f1f080ac370 con 0x7f1f081405a0 2024-03-29T11:58:30.685 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.671+0000 7f1efdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:58:30.686 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.671+0000 7f1efdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:58:30.686 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.671+0000 7f1efdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:58:30.686 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.671+0000 7f1efdffb640 10 monclient(hunting): my global_id is 4622 2024-03-29T11:58:30.686 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.671+0000 7f1efdffb640 1 -- 172.21.15.98:0/1038734959 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f0812f3b0 con 0x7f1f081405a0 2024-03-29T11:58:30.686 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.671+0000 7f1efdffb640 1 -- 172.21.15.98:0/1038734959 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 523202880 0 0) 0x7f1f080ac090 con 0x7f1f0806ade0 2024-03-29T11:58:30.686 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.671+0000 7f1efdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:58:30.686 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.671+0000 7f1efdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:58:30.686 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.671+0000 7f1efdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:58:30.686 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.671+0000 7f1efdffb640 10 monclient(hunting): my global_id is 4570 2024-03-29T11:58:30.686 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.671+0000 7f1efdffb640 1 -- 172.21.15.98:0/1038734959 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f080ac370 con 0x7f1f0806ade0 2024-03-29T11:58:30.686 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.671+0000 7f1efdffb640 1 -- 172.21.15.98:0/1038734959 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3244101118 0 0) 0x7f1eec004eb0 con 0x7f1f0813d830 2024-03-29T11:58:30.686 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.671+0000 7f1efdffb640 1 -- 172.21.15.98:0/1038734959 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f080ac090 con 0x7f1f0813d830 2024-03-29T11:58:30.686 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.671+0000 7f1efdffb640 1 -- 172.21.15.98:0/1038734959 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2627117629 0 0) 0x7f1f0812f3b0 con 0x7f1f081405a0 2024-03-29T11:58:30.687 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.671+0000 7f1efdffb640 1 -- 172.21.15.98:0/1038734959 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1eec004eb0 con 0x7f1f081405a0 2024-03-29T11:58:30.687 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.671+0000 7f1efdffb640 1 -- 172.21.15.98:0/1038734959 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f1ef8001870 con 0x7f1f0813d830 2024-03-29T11:58:30.687 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.671+0000 7f1efdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:58:30.687 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.671+0000 7f1efdffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:58:30.687 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.671+0000 7f1efdffb640 10 monclient(hunting): dump: 2024-03-29T11:58:30.687 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:58:30.687 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:30.687 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:30.687 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:30.687 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:58:30.687 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:30.687 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:58:30.687 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:58:30.687 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:58:30.688 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:30.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.671+0000 7f1efdffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:58:30.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.671+0000 7f1efdffb640 1 -- 172.21.15.98:0/1038734959 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f1f00001ee0 con 0x7f1f081405a0 2024-03-29T11:58:30.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.671+0000 7f1efdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:58:30.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.671+0000 7f1efdffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:58:30.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.671+0000 7f1efdffb640 10 monclient(hunting): dump: 2024-03-29T11:58:30.688 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:58:30.688 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:30.688 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:30.688 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:30.688 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:58:30.688 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:30.688 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:58:30.688 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:58:30.688 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:58:30.689 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:30.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.671+0000 7f1efdffb640 1 -- 172.21.15.98:0/1038734959 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2199341981 0 0) 0x7f1f080ac370 con 0x7f1f0806ade0 2024-03-29T11:58:30.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.671+0000 7f1efdffb640 1 -- 172.21.15.98:0/1038734959 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f0812f3b0 con 0x7f1f0806ade0 2024-03-29T11:58:30.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.671+0000 7f1efdffb640 1 -- 172.21.15.98:0/1038734959 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f1f04001870 con 0x7f1f0806ade0 2024-03-29T11:58:30.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.671+0000 7f1efdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:58:30.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.671+0000 7f1efdffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:58:30.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.672+0000 7f1efdffb640 10 monclient(hunting): dump: 2024-03-29T11:58:30.689 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:58:30.689 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:30.689 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:30.689 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:30.689 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:58:30.689 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:30.689 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:58:30.690 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:58:30.690 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:58:30.690 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:30.690 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.672+0000 7f1efdffb640 1 -- 172.21.15.98:0/1038734959 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3502383148 0 0) 0x7f1f080ac090 con 0x7f1f0813d830 2024-03-29T11:58:30.690 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.672+0000 7f1efdffb640 1 -- 172.21.15.98:0/1038734959 >> v1:172.21.15.98:6790/0 conn(0x7f1f081405a0 legacy=0x7f1f081e1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:30.690 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.672+0000 7f1efdffb640 1 -- 172.21.15.98:0/1038734959 >> v1:172.21.15.121:6789/0 conn(0x7f1f0806ade0 legacy=0x7f1f080aa840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:30.690 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.672+0000 7f1efdffb640 10 monclient: _finish_hunting 0 2024-03-29T11:58:30.690 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.672+0000 7f1efdffb640 1 monclient: found mon.a 2024-03-29T11:58:30.690 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.672+0000 7f1efdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:58:30.690 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.672+0000 7f1efdffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:58:30.690 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.672+0000 7f1efdffb640 1 -- 172.21.15.98:0/1038734959 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f0812efb0 con 0x7f1f0813d830 2024-03-29T11:58:30.690 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.672+0000 7f1efdffb640 10 monclient: _finish_auth 0 2024-03-29T11:58:30.690 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.672+0000 7f1efdffb640 10 monclient: _check_auth_tickets 2024-03-29T11:58:30.690 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.672+0000 7f1efdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:58:30.690 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.672+0000 7f1f0f8a8640 5 monclient: authenticate success, global_id 4581 2024-03-29T11:58:30.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.672+0000 7f1f0f8a8640 10 monclient: _renew_subs 2024-03-29T11:58:30.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.672+0000 7f1f0f8a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:58:30.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.672+0000 7f1f0f8a8640 1 -- 172.21.15.98:0/1038734959 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f080aa3b0 con 0x7f1f0813d830 2024-03-29T11:58:30.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.673+0000 7f1f0f8a8640 10 client.4581.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:58:30.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.673+0000 7f1f0f8a8640 10 monclient: _renew_subs 2024-03-29T11:58:30.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.673+0000 7f1f0f8a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:58:30.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.673+0000 7f1f0f8a8640 1 -- 172.21.15.98:0/1038734959 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f080b3b70 con 0x7f1f0813d830 2024-03-29T11:58:30.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.673+0000 7f1f0f8a8640 1 librados: init done 2024-03-29T11:58:30.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.673+0000 7f1efdffb640 1 -- 172.21.15.98:0/1038734959 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ef8001d80 con 0x7f1f0813d830 2024-03-29T11:58:30.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.673+0000 7f1efdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:58:30.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.673+0000 7f1efdffb640 1 -- 172.21.15.98:0/1038734959 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f1ef8003830 con 0x7f1f0813d830 2024-03-29T11:58:30.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.673+0000 7f1efdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:58:30.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.673+0000 7f1efdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:58:30.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.673+0000 7f1efdffb640 10 monclient: dump: 2024-03-29T11:58:30.691 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:58:30.692 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:30.692 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:30.692 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:30.692 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:58:30.692 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:30.692 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:58:30.692 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:58:30.692 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:58:30.692 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:30.692 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.675+0000 7f1efeffd640 4 set_mon_vals no callback set 2024-03-29T11:58:30.692 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.675+0000 7f1efdffb640 1 -- 172.21.15.98:0/1038734959 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f1ef8003990 con 0x7f1f0813d830 2024-03-29T11:58:30.692 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.676+0000 7f1f0f8a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:58:30.692 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.676+0000 7f1efdffb640 1 -- 172.21.15.98:0/1038734959 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3033626086 0 0) 0x7f1ef80837f0 con 0x7f1f0813d830 2024-03-29T11:58:30.692 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.676+0000 7f1efdffb640 10 client.4581.objecter ms_dispatch 0x7f1f08000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:58:30.692 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.676+0000 7f1efdffb640 3 client.4581.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:58:30.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.676+0000 7f1efdffb640 3 client.4581.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:58:30.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.676+0000 7f1efdffb640 20 client.4581.objecter dump_active .. 0 homeless 2024-03-29T11:58:30.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.676+0000 7f1f0f8a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:58:30.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.677+0000 7f1f0f8a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:58:30.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.677+0000 7f1f0f8a8640 1 -- 172.21.15.98:0/1038734959 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ee8005080 con 0x7f1f0813d830 2024-03-29T11:58:30.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.679+0000 7f1f0d644640 10 monclient: get_auth_request con 0x7f1eec06e0b0 auth_method 0 2024-03-29T11:58:30.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.679+0000 7f1efdffb640 10 client.4581.objecter ms_handle_connect 0x7f1eec06e0b0 2024-03-29T11:58:30.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.686+0000 7f1efdffb640 1 -- 172.21.15.98:0/1038734959 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1ef804ef00 con 0x7f1f0813d830 2024-03-29T11:58:30.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.686+0000 7f1efdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:58:30.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:30.686+0000 7f1efdffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:58:31.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:31.002+0000 7f1f0f8a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:58:31.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:31.002+0000 7f1f0f8a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:58:31.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:31.002+0000 7f1f0f8a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:58:31.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:31.002+0000 7f1f0f8a8640 1 -- 172.21.15.98:0/1038734959 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ee8005330 con 0x7f1f0813d830 2024-03-29T11:58:31.011 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:31.004+0000 7f1efdffb640 1 -- 172.21.15.98:0/1038734959 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3141891755) 0x7f1ef8051000 con 0x7f1f0813d830 2024-03-29T11:58:31.011 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:31.004+0000 7f1efdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:58:31.011 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:31.004+0000 7f1efdffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:58:31.012 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:58:31.012 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":11,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:58:22.774801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:58:22.774801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:58:31.015 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:31.008+0000 7f1f0f8a8640 10 librados: watch_flush enter 2024-03-29T11:58:31.015 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:31.008+0000 7f1f0f8a8640 10 librados: watch_flush exit 2024-03-29T11:58:31.015 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:31.008+0000 7f1f0f8a8640 20 client.4581.objecter shutdown clearing up homeless session... 2024-03-29T11:58:31.015 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:31.008+0000 7f1f0f8a8640 10 client.4581.objecter successfully canceled tick 2024-03-29T11:58:31.016 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:31.009+0000 7f1f0f8a8640 1 -- 172.21.15.98:0/1038734959 >> v1:172.21.15.98:6816/4065186974 conn(0x7f1eec06e0b0 legacy=0x7f1eec070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:31.016 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:31.009+0000 7f1f0f8a8640 10 monclient: shutdown 2024-03-29T11:58:31.017 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:31.009+0000 7f1f0f8a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:58:31.017 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:31.009+0000 7f1f0f8a8640 1 -- 172.21.15.98:0/1038734959 >> v1:172.21.15.98:6789/0 conn(0x7f1f0813d830 legacy=0x7f1f080a9ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:31.017 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:31.009+0000 7f1f0f8a8640 1 -- 172.21.15.98:0/1038734959 shutdown_connections 2024-03-29T11:58:31.017 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:31.009+0000 7f1f0f8a8640 1 -- 172.21.15.98:0/1038734959 >> 172.21.15.98:0/1038734959 conn(0x7f1f080b4930 msgr2=0x7f1f08138c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:58:31.017 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:31.009+0000 7f1f0f8a8640 1 -- 172.21.15.98:0/1038734959 shutdown_connections 2024-03-29T11:58:31.017 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:31.009+0000 7f1f0f8a8640 1 -- 172.21.15.98:0/1038734959 wait complete. 2024-03-29T11:58:31.017 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:31.009+0000 7f1f0f8a8640 1 librados: shutdown 2024-03-29T11:58:31.033 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:58:37.841 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:58:37.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.981+0000 7fbef0523640 10 monclient: get_monmap_and_config 2024-03-29T11:58:37.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.981+0000 7fbef0523640 10 monclient: build_initial_monmap 2024-03-29T11:58:37.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.981+0000 7fbef0523640 10 monclient: monmap: 2024-03-29T11:58:37.989 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:58:37.989 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:37.989 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:58:37.982729+0000 2024-03-29T11:58:37.989 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:58:37.982729+0000 2024-03-29T11:58:37.989 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:58:37.989 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:37.989 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:58:37.989 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:58:37.990 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:58:37.990 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:37.991 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.984+0000 7fbef0523640 1 Processor -- start 2024-03-29T11:58:37.994 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.986+0000 7fbef0523640 1 -- start start 2024-03-29T11:58:37.994 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.986+0000 7fbef0523640 10 monclient: init 2024-03-29T11:58:37.994 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.988+0000 7fbef0523640 10 monclient: _reopen_session rank -1 2024-03-29T11:58:37.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.988+0000 7fbef0523640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:58:37.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.988+0000 7fbef0523640 10 monclient(hunting): picked mon.noname-a con 0x7fbee80b0770 addr v1:172.21.15.98:6789/0 2024-03-29T11:58:37.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.988+0000 7fbef0523640 10 monclient(hunting): picked mon.noname-c con 0x7fbee814a700 addr v1:172.21.15.121:6789/0 2024-03-29T11:58:37.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.988+0000 7fbef0523640 10 monclient(hunting): picked mon.noname-b con 0x7fbee80ac5a0 addr v1:172.21.15.98:6790/0 2024-03-29T11:58:37.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.988+0000 7fbef0523640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbee80aba70 con 0x7fbee80b0770 2024-03-29T11:58:37.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.988+0000 7fbef0523640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbee80aecb0 con 0x7fbee814a700 2024-03-29T11:58:37.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.988+0000 7fbef0523640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbee80af150 con 0x7fbee80ac5a0 2024-03-29T11:58:37.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.988+0000 7fbef0523640 10 monclient(hunting): _renew_subs 2024-03-29T11:58:37.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.988+0000 7fbef0523640 10 monclient(hunting): authenticate will time out at 1425.073324s 2024-03-29T11:58:37.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.988+0000 7fbeeeac0640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fbee80ac5a0 0x7fbee8151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:59328/0 (socket says 172.21.15.98:59328) 2024-03-29T11:58:37.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.988+0000 7fbeeeac0640 1 -- 172.21.15.98:0/186221865 learned_addr learned my addr 172.21.15.98:0/186221865 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:58:37.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.989+0000 7fbeed2bd640 1 -- 172.21.15.98:0/186221865 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2784538917 0 0) 0x7fbee80af150 con 0x7fbee80ac5a0 2024-03-29T11:58:37.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.989+0000 7fbeed2bd640 10 monclient(hunting): _negotiate 2024-03-29T11:58:37.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.989+0000 7fbeed2bd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:58:37.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.989+0000 7fbeed2bd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:58:37.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.989+0000 7fbeed2bd640 10 monclient(hunting): my global_id is 4625 2024-03-29T11:58:37.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.989+0000 7fbeed2bd640 1 -- 172.21.15.98:0/186221865 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbed4003520 con 0x7fbee80ac5a0 2024-03-29T11:58:38.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.990+0000 7fbeed2bd640 1 -- 172.21.15.98:0/186221865 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1803476908 0 0) 0x7fbed4003520 con 0x7fbee80ac5a0 2024-03-29T11:58:38.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.990+0000 7fbeed2bd640 1 -- 172.21.15.98:0/186221865 >> v1:172.21.15.121:6789/0 conn(0x7fbee814a700 legacy=0x7fbee80abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:58:38.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.990+0000 7fbeed2bd640 1 -- 172.21.15.98:0/186221865 >> v1:172.21.15.98:6789/0 conn(0x7fbee80b0770 legacy=0x7fbee8149ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:58:38.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.990+0000 7fbeed2bd640 10 monclient: _finish_hunting 0 2024-03-29T11:58:38.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.990+0000 7fbeed2bd640 1 monclient: found mon.noname-b 2024-03-29T11:58:38.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.990+0000 7fbeed2bd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:58:38.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.990+0000 7fbeed2bd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:58:38.002 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.990+0000 7fbeed2bd640 1 -- 172.21.15.98:0/186221865 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbee80ac9d0 con 0x7fbee80ac5a0 2024-03-29T11:58:38.002 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.990+0000 7fbeed2bd640 10 monclient: _finish_auth 0 2024-03-29T11:58:38.002 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.990+0000 7fbeed2bd640 10 monclient: _check_auth_tickets 2024-03-29T11:58:38.002 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.990+0000 7fbeed2bd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:58:38.002 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.990+0000 7fbef0523640 5 monclient: authenticate success, global_id 4625 2024-03-29T11:58:38.002 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.990+0000 7fbef0523640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:58:38.002 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.991+0000 7fbeed2bd640 1 -- 172.21.15.98:0/186221865 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fbee0001a10 con 0x7fbee80ac5a0 2024-03-29T11:58:38.002 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.991+0000 7fbeed2bd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:58:38.002 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.991+0000 7fbeed2bd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:58:38.002 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.991+0000 7fbeed2bd640 10 monclient: dump: 2024-03-29T11:58:38.002 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:58:38.002 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:38.002 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:38.002 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:38.003 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:58:38.003 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:38.003 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:58:38.003 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:58:38.004 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:58:38.004 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:38.004 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.991+0000 7fbef0523640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:58:38.004 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.991+0000 7fbeed2bd640 1 -- 172.21.15.98:0/186221865 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbee0001ca0 con 0x7fbee80ac5a0 2024-03-29T11:58:38.004 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.991+0000 7fbeed2bd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:58:38.004 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.991+0000 7fbef0523640 10 monclient: get_monmap_and_config success 2024-03-29T11:58:38.004 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.991+0000 7fbef0523640 4 set_mon_vals no callback set 2024-03-29T11:58:38.004 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.992+0000 7fbef0523640 10 monclient: shutdown 2024-03-29T11:58:38.004 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.992+0000 7fbef0523640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:58:38.004 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.992+0000 7fbef0523640 1 -- 172.21.15.98:0/186221865 >> v1:172.21.15.98:6790/0 conn(0x7fbee80ac5a0 legacy=0x7fbee8151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:38.004 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.994+0000 7fbef0523640 1 -- 172.21.15.98:0/186221865 shutdown_connections 2024-03-29T11:58:38.004 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.994+0000 7fbef0523640 1 -- 172.21.15.98:0/186221865 >> 172.21.15.98:0/186221865 conn(0x7fbee806b660 msgr2=0x7fbee806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:58:38.005 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.995+0000 7fbef0523640 1 -- 172.21.15.98:0/186221865 shutdown_connections 2024-03-29T11:58:38.005 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.996+0000 7fbef0523640 1 -- 172.21.15.98:0/186221865 wait complete. 2024-03-29T11:58:38.005 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.996+0000 7fbef0523640 10 monclient: build_initial_monmap 2024-03-29T11:58:38.005 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.996+0000 7fbef0523640 10 monclient: monmap: 2024-03-29T11:58:38.005 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:58:38.005 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:58:38.006 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:58:38.006 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:58:38.006 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:58:38.006 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:38.006 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:58:38.006 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:58:38.006 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:58:38.006 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:38.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.997+0000 7fbef0523640 1 librados: starting msgr at 2024-03-29T11:58:38.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.997+0000 7fbef0523640 1 librados: starting objecter 2024-03-29T11:58:38.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.997+0000 7fbef0523640 1 Processor -- start 2024-03-29T11:58:38.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.997+0000 7fbef0523640 1 -- start start 2024-03-29T11:58:38.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.997+0000 7fbef0523640 1 librados: setting wanted keys 2024-03-29T11:58:38.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.997+0000 7fbef0523640 1 librados: calling monclient init 2024-03-29T11:58:38.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.997+0000 7fbef0523640 10 monclient: init 2024-03-29T11:58:38.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.998+0000 7fbef0523640 10 monclient: _reopen_session rank -1 2024-03-29T11:58:38.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.998+0000 7fbef0523640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:58:38.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.998+0000 7fbef0523640 10 monclient(hunting): picked mon.noname-b con 0x7fbee80ac5a0 addr v1:172.21.15.121:6789/0 2024-03-29T11:58:38.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.998+0000 7fbef0523640 10 monclient(hunting): picked mon.noname-a con 0x7fbee80b0770 addr v1:172.21.15.98:6789/0 2024-03-29T11:58:38.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.998+0000 7fbef0523640 10 monclient(hunting): picked mon.noname-c con 0x7fbee814a700 addr v1:172.21.15.98:6790/0 2024-03-29T11:58:38.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.998+0000 7fbef0523640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbee811ee10 con 0x7fbee80b0770 2024-03-29T11:58:38.008 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.998+0000 7fbef0523640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbee811d520 con 0x7fbee80ac5a0 2024-03-29T11:58:38.008 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.998+0000 7fbef0523640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbee811da20 con 0x7fbee814a700 2024-03-29T11:58:38.008 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.998+0000 7fbef0523640 10 monclient(hunting): _renew_subs 2024-03-29T11:58:38.008 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:37.998+0000 7fbef0523640 10 monclient(hunting): authenticate will time out at 1425.083444s 2024-03-29T11:58:38.008 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.000+0000 7fbeeeac0640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fbee814a700 0x7fbee8122d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:33222/0 (socket says 172.21.15.98:33222) 2024-03-29T11:58:38.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.000+0000 7fbeeeac0640 1 -- 172.21.15.98:0/3275037927 learned_addr learned my addr 172.21.15.98:0/3275037927 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:58:38.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.001+0000 7fbedeffd640 10 client.?.objecter ms_handle_connect 0x7fbee814a700 2024-03-29T11:58:38.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.001+0000 7fbedeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:58:38.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.001+0000 7fbedeffd640 10 client.?.objecter ms_handle_connect 0x7fbee80ac5a0 2024-03-29T11:58:38.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.001+0000 7fbedeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:58:38.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.001+0000 7fbedeffd640 1 -- 172.21.15.98:0/3275037927 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 197693941 0 0) 0x7fbee811da20 con 0x7fbee814a700 2024-03-29T11:58:38.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.001+0000 7fbedeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:58:38.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.001+0000 7fbedeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:58:38.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.001+0000 7fbedeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:58:38.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.001+0000 7fbedeffd640 10 monclient(hunting): my global_id is 4628 2024-03-29T11:58:38.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.001+0000 7fbedeffd640 1 -- 172.21.15.98:0/3275037927 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbecc004eb0 con 0x7fbee814a700 2024-03-29T11:58:38.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.001+0000 7fbedeffd640 1 -- 172.21.15.98:0/3275037927 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1820277615 0 0) 0x7fbee811d520 con 0x7fbee80ac5a0 2024-03-29T11:58:38.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.001+0000 7fbedeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:58:38.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.001+0000 7fbedeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:58:38.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.001+0000 7fbedeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:58:38.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.001+0000 7fbedeffd640 10 monclient(hunting): my global_id is 4573 2024-03-29T11:58:38.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.001+0000 7fbedeffd640 1 -- 172.21.15.98:0/3275037927 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbee811da20 con 0x7fbee80ac5a0 2024-03-29T11:58:38.013 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.003+0000 7fbedeffd640 1 -- 172.21.15.98:0/3275037927 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2669129665 0 0) 0x7fbecc004eb0 con 0x7fbee814a700 2024-03-29T11:58:38.013 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.003+0000 7fbedeffd640 1 -- 172.21.15.98:0/3275037927 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbee811d520 con 0x7fbee814a700 2024-03-29T11:58:38.013 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.003+0000 7fbedeffd640 1 -- 172.21.15.98:0/3275037927 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fbee0004e60 con 0x7fbee814a700 2024-03-29T11:58:38.013 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.003+0000 7fbedeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:58:38.013 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.003+0000 7fbedeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:58:38.014 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.003+0000 7fbedeffd640 10 monclient(hunting): dump: 2024-03-29T11:58:38.014 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:58:38.014 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:38.014 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:38.014 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:38.014 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:58:38.014 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:38.014 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:58:38.014 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:58:38.014 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:58:38.015 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:38.015 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.003+0000 7fbedeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:58:38.015 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.004+0000 7fbedeffd640 10 client.?.objecter ms_handle_connect 0x7fbee80b0770 2024-03-29T11:58:38.015 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.004+0000 7fbedeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:58:38.015 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.004+0000 7fbedeffd640 1 -- 172.21.15.98:0/3275037927 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2369424070 0 0) 0x7fbee811ee10 con 0x7fbee80b0770 2024-03-29T11:58:38.015 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.004+0000 7fbedeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:58:38.015 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.004+0000 7fbedeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:58:38.015 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.004+0000 7fbedeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:58:38.015 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.004+0000 7fbedeffd640 10 monclient(hunting): my global_id is 4584 2024-03-29T11:58:38.015 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.004+0000 7fbedeffd640 1 -- 172.21.15.98:0/3275037927 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbecc004eb0 con 0x7fbee80b0770 2024-03-29T11:58:38.015 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.005+0000 7fbedeffd640 1 -- 172.21.15.98:0/3275037927 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2183017295 0 0) 0x7fbecc004eb0 con 0x7fbee80b0770 2024-03-29T11:58:38.015 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.005+0000 7fbedeffd640 1 -- 172.21.15.98:0/3275037927 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbee811ee10 con 0x7fbee80b0770 2024-03-29T11:58:38.015 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.005+0000 7fbedeffd640 1 -- 172.21.15.98:0/3275037927 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fbed8001700 con 0x7fbee80b0770 2024-03-29T11:58:38.015 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.005+0000 7fbedeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:58:38.016 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.005+0000 7fbedeffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:58:38.016 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.005+0000 7fbedeffd640 10 monclient(hunting): dump: 2024-03-29T11:58:38.016 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:58:38.016 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:38.016 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:38.016 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:38.016 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:58:38.016 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:38.016 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:58:38.016 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:58:38.016 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:58:38.016 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:38.016 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.005+0000 7fbedeffd640 1 -- 172.21.15.98:0/3275037927 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1256459035 0 0) 0x7fbee811ee10 con 0x7fbee80b0770 2024-03-29T11:58:38.017 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.005+0000 7fbedeffd640 1 -- 172.21.15.98:0/3275037927 >> v1:172.21.15.98:6790/0 conn(0x7fbee814a700 legacy=0x7fbee8122d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:38.017 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.005+0000 7fbedeffd640 1 -- 172.21.15.98:0/3275037927 >> v1:172.21.15.121:6789/0 conn(0x7fbee80ac5a0 legacy=0x7fbee81e3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:38.017 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.005+0000 7fbedeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:58:38.017 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.005+0000 7fbedeffd640 1 monclient: found mon.a 2024-03-29T11:58:38.017 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.005+0000 7fbedeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:58:38.017 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.005+0000 7fbedeffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:58:38.017 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.005+0000 7fbedeffd640 1 -- 172.21.15.98:0/3275037927 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbee811dc90 con 0x7fbee80b0770 2024-03-29T11:58:38.017 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.005+0000 7fbedeffd640 10 monclient: _finish_auth 0 2024-03-29T11:58:38.017 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.005+0000 7fbedeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:58:38.017 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.005+0000 7fbedeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:58:38.017 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.005+0000 7fbef0523640 5 monclient: authenticate success, global_id 4584 2024-03-29T11:58:38.017 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.005+0000 7fbef0523640 10 monclient: _renew_subs 2024-03-29T11:58:38.017 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.005+0000 7fbef0523640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:58:38.018 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.005+0000 7fbef0523640 1 -- 172.21.15.98:0/3275037927 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbee811d090 con 0x7fbee80b0770 2024-03-29T11:58:38.018 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.005+0000 7fbef0523640 10 client.4584.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:58:38.018 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.005+0000 7fbef0523640 10 monclient: _renew_subs 2024-03-29T11:58:38.018 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.005+0000 7fbef0523640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:58:38.018 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.005+0000 7fbef0523640 1 -- 172.21.15.98:0/3275037927 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbee8070de0 con 0x7fbee80b0770 2024-03-29T11:58:38.018 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.005+0000 7fbef0523640 1 librados: init done 2024-03-29T11:58:38.018 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.008+0000 7fbedeffd640 1 -- 172.21.15.98:0/3275037927 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbed8001bf0 con 0x7fbee80b0770 2024-03-29T11:58:38.019 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.008+0000 7fbedeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:58:38.019 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.008+0000 7fbedeffd640 1 -- 172.21.15.98:0/3275037927 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fbed80037a0 con 0x7fbee80b0770 2024-03-29T11:58:38.019 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.008+0000 7fbedeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:58:38.019 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.008+0000 7fbedeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:58:38.019 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.008+0000 7fbedeffd640 10 monclient: dump: 2024-03-29T11:58:38.019 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:58:38.019 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:38.019 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:38.019 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:38.019 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:58:38.019 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:38.019 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:58:38.019 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:58:38.019 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:58:38.020 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:38.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.008+0000 7fbef0523640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:58:38.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.008+0000 7fbedeffd640 1 -- 172.21.15.98:0/3275037927 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7fbed8019770 con 0x7fbee80b0770 2024-03-29T11:58:38.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.008+0000 7fbef0523640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:58:38.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.008+0000 7fbef0523640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:58:38.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.008+0000 7fbef0523640 1 -- 172.21.15.98:0/3275037927 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbee80ab560 con 0x7fbee80b0770 2024-03-29T11:58:38.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.008+0000 7fbedffff640 4 set_mon_vals no callback set 2024-03-29T11:58:38.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.009+0000 7fbedeffd640 1 -- 172.21.15.98:0/3275037927 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3033626086 0 0) 0x7fbed8083750 con 0x7fbee80b0770 2024-03-29T11:58:38.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.009+0000 7fbedeffd640 10 client.4584.objecter ms_dispatch 0x7fbee8000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:58:38.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.009+0000 7fbedeffd640 3 client.4584.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:58:38.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.009+0000 7fbedeffd640 3 client.4584.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:58:38.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.009+0000 7fbedeffd640 20 client.4584.objecter dump_active .. 0 homeless 2024-03-29T11:58:38.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.009+0000 7fbeee2bf640 10 monclient: get_auth_request con 0x7fbecc06e100 auth_method 0 2024-03-29T11:58:38.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.010+0000 7fbedeffd640 10 client.4584.objecter ms_handle_connect 0x7fbecc06e100 2024-03-29T11:58:38.027 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.019+0000 7fbedeffd640 1 -- 172.21.15.98:0/3275037927 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbed804f0b0 con 0x7fbee80b0770 2024-03-29T11:58:38.027 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.019+0000 7fbedeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:58:38.028 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.019+0000 7fbedeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:58:38.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.337+0000 7fbef0523640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:58:38.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.337+0000 7fbef0523640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:58:38.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.337+0000 7fbef0523640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:58:38.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.337+0000 7fbef0523640 1 -- 172.21.15.98:0/3275037927 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbee811d6b0 con 0x7fbee80b0770 2024-03-29T11:58:38.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.338+0000 7fbedeffd640 1 -- 172.21.15.98:0/3275037927 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3141891755) 0x7fbed8050fe0 con 0x7fbee80b0770 2024-03-29T11:58:38.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.338+0000 7fbedeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:58:38.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.338+0000 7fbedeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:58:38.347 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:58:38.347 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":11,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:58:22.774801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:58:22.774801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:58:38.349 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.341+0000 7fbef0523640 10 librados: watch_flush enter 2024-03-29T11:58:38.349 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.341+0000 7fbef0523640 10 librados: watch_flush exit 2024-03-29T11:58:38.349 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.341+0000 7fbef0523640 20 client.4584.objecter shutdown clearing up homeless session... 2024-03-29T11:58:38.349 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.341+0000 7fbef0523640 10 client.4584.objecter successfully canceled tick 2024-03-29T11:58:38.349 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.342+0000 7fbef0523640 1 -- 172.21.15.98:0/3275037927 >> v1:172.21.15.98:6816/4065186974 conn(0x7fbecc06e100 legacy=0x7fbecc0705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:38.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.342+0000 7fbef0523640 10 monclient: shutdown 2024-03-29T11:58:38.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.342+0000 7fbef0523640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:58:38.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.342+0000 7fbef0523640 1 -- 172.21.15.98:0/3275037927 >> v1:172.21.15.98:6789/0 conn(0x7fbee80b0770 legacy=0x7fbee8122640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:38.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.342+0000 7fbef0523640 1 -- 172.21.15.98:0/3275037927 shutdown_connections 2024-03-29T11:58:38.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.342+0000 7fbef0523640 1 -- 172.21.15.98:0/3275037927 >> 172.21.15.98:0/3275037927 conn(0x7fbee806b660 msgr2=0x7fbee80abe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:58:38.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.342+0000 7fbef0523640 1 -- 172.21.15.98:0/3275037927 shutdown_connections 2024-03-29T11:58:38.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.342+0000 7fbef0523640 1 -- 172.21.15.98:0/3275037927 wait complete. 2024-03-29T11:58:38.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:38.342+0000 7fbef0523640 1 librados: shutdown 2024-03-29T11:58:38.371 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:58:45.180 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:58:45.326 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.319+0000 7f6d795cf640 10 monclient: get_monmap_and_config 2024-03-29T11:58:45.326 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.319+0000 7f6d795cf640 10 monclient: build_initial_monmap 2024-03-29T11:58:45.327 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.319+0000 7f6d795cf640 10 monclient: monmap: 2024-03-29T11:58:45.327 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:58:45.327 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:45.327 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:58:45.320448+0000 2024-03-29T11:58:45.327 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:58:45.320448+0000 2024-03-29T11:58:45.327 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:58:45.327 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:45.327 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:58:45.327 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:58:45.327 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:58:45.327 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:45.328 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.320+0000 7f6d795cf640 1 Processor -- start 2024-03-29T11:58:45.328 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.320+0000 7f6d795cf640 1 -- start start 2024-03-29T11:58:45.328 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.320+0000 7f6d795cf640 10 monclient: init 2024-03-29T11:58:45.328 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.321+0000 7f6d795cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:58:45.328 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.321+0000 7f6d795cf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:58:45.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.321+0000 7f6d795cf640 10 monclient(hunting): picked mon.noname-a con 0x7f6d740b4890 addr v1:172.21.15.98:6789/0 2024-03-29T11:58:45.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.321+0000 7f6d795cf640 10 monclient(hunting): picked mon.noname-c con 0x7f6d7413c040 addr v1:172.21.15.121:6789/0 2024-03-29T11:58:45.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.321+0000 7f6d795cf640 10 monclient(hunting): picked mon.noname-b con 0x7f6d741427e0 addr v1:172.21.15.98:6790/0 2024-03-29T11:58:45.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.321+0000 7f6d737fe640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f6d741427e0 0x7f6d7414d880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:54594/0 (socket says 172.21.15.98:54594) 2024-03-29T11:58:45.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.321+0000 7f6d737fe640 1 -- 172.21.15.98:0/3503441533 learned_addr learned my addr 172.21.15.98:0/3503441533 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:58:45.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.321+0000 7f6d795cf640 1 -- 172.21.15.98:0/3503441533 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d740b42f0 con 0x7f6d740b4890 2024-03-29T11:58:45.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.321+0000 7f6d795cf640 1 -- 172.21.15.98:0/3503441533 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d740aa7a0 con 0x7f6d7413c040 2024-03-29T11:58:45.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.321+0000 7f6d795cf640 1 -- 172.21.15.98:0/3503441533 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d740aaa10 con 0x7f6d741427e0 2024-03-29T11:58:45.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.322+0000 7f6d795cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:58:45.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.322+0000 7f6d795cf640 10 monclient(hunting): authenticate will time out at 1432.407077s 2024-03-29T11:58:45.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.322+0000 7f6d71ffb640 1 -- 172.21.15.98:0/3503441533 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 667896267 0 0) 0x7f6d740aaa10 con 0x7f6d741427e0 2024-03-29T11:58:45.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.322+0000 7f6d71ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:58:45.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.322+0000 7f6d71ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:58:45.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.322+0000 7f6d71ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:58:45.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.322+0000 7f6d71ffb640 10 monclient(hunting): my global_id is 4631 2024-03-29T11:58:45.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.322+0000 7f6d71ffb640 1 -- 172.21.15.98:0/3503441533 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d5c003520 con 0x7f6d741427e0 2024-03-29T11:58:45.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.322+0000 7f6d71ffb640 1 -- 172.21.15.98:0/3503441533 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1339096390 0 0) 0x7f6d740aa7a0 con 0x7f6d7413c040 2024-03-29T11:58:45.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.322+0000 7f6d71ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:58:45.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.322+0000 7f6d71ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:58:45.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.322+0000 7f6d71ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:58:45.334 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.322+0000 7f6d71ffb640 10 monclient(hunting): my global_id is 4576 2024-03-29T11:58:45.334 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.322+0000 7f6d71ffb640 1 -- 172.21.15.98:0/3503441533 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d740aaa10 con 0x7f6d7413c040 2024-03-29T11:58:45.334 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.323+0000 7f6d71ffb640 1 -- 172.21.15.98:0/3503441533 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2125571712 0 0) 0x7f6d5c003520 con 0x7f6d741427e0 2024-03-29T11:58:45.334 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.323+0000 7f6d71ffb640 1 -- 172.21.15.98:0/3503441533 >> v1:172.21.15.121:6789/0 conn(0x7f6d7413c040 legacy=0x7f6d741420d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:45.334 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.323+0000 7f6d71ffb640 1 -- 172.21.15.98:0/3503441533 >> v1:172.21.15.98:6789/0 conn(0x7f6d740b4890 legacy=0x7f6d7413b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:58:45.334 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.323+0000 7f6d71ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:58:45.334 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.323+0000 7f6d71ffb640 1 monclient: found mon.noname-b 2024-03-29T11:58:45.334 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.323+0000 7f6d71ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:58:45.334 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.323+0000 7f6d71ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:58:45.335 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.323+0000 7f6d71ffb640 1 -- 172.21.15.98:0/3503441533 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d740aacf0 con 0x7f6d741427e0 2024-03-29T11:58:45.335 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.323+0000 7f6d71ffb640 10 monclient: _finish_auth 0 2024-03-29T11:58:45.335 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.323+0000 7f6d71ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:58:45.335 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.323+0000 7f6d71ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:58:45.335 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.323+0000 7f6d795cf640 5 monclient: authenticate success, global_id 4631 2024-03-29T11:58:45.335 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.323+0000 7f6d795cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:58:45.335 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.323+0000 7f6d71ffb640 1 -- 172.21.15.98:0/3503441533 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f6d6c001a10 con 0x7f6d741427e0 2024-03-29T11:58:45.335 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.323+0000 7f6d71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:58:45.335 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.323+0000 7f6d71ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:58:45.336 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.323+0000 7f6d71ffb640 10 monclient: dump: 2024-03-29T11:58:45.336 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:58:45.336 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:45.336 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:45.336 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:45.336 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:58:45.336 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:45.336 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:58:45.336 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:58:45.336 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:58:45.336 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:45.336 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.323+0000 7f6d795cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:58:45.336 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.323+0000 7f6d71ffb640 1 -- 172.21.15.98:0/3503441533 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d6c001bf0 con 0x7f6d741427e0 2024-03-29T11:58:45.337 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.323+0000 7f6d71ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:58:45.337 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.323+0000 7f6d71ffb640 1 -- 172.21.15.98:0/3503441533 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f6d6c001f80 con 0x7f6d741427e0 2024-03-29T11:58:45.337 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.323+0000 7f6d71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:58:45.337 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.323+0000 7f6d71ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:58:45.337 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.323+0000 7f6d71ffb640 10 monclient: dump: 2024-03-29T11:58:45.337 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:58:45.337 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:45.337 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:45.337 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:45.337 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:58:45.337 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:45.337 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:58:45.337 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:58:45.337 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:58:45.338 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:45.338 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.323+0000 7f6d795cf640 10 monclient: get_monmap_and_config success 2024-03-29T11:58:45.338 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.323+0000 7f6d795cf640 4 set_mon_vals no callback set 2024-03-29T11:58:45.338 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.324+0000 7f6d795cf640 10 monclient: shutdown 2024-03-29T11:58:45.338 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.324+0000 7f6d795cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:58:45.338 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.324+0000 7f6d795cf640 1 -- 172.21.15.98:0/3503441533 >> v1:172.21.15.98:6790/0 conn(0x7f6d741427e0 legacy=0x7f6d7414d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:45.338 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.325+0000 7f6d795cf640 1 -- 172.21.15.98:0/3503441533 shutdown_connections 2024-03-29T11:58:45.338 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.325+0000 7f6d795cf640 1 -- 172.21.15.98:0/3503441533 >> 172.21.15.98:0/3503441533 conn(0x7f6d7406d180 msgr2=0x7f6d74066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:58:45.338 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.325+0000 7f6d795cf640 1 -- 172.21.15.98:0/3503441533 shutdown_connections 2024-03-29T11:58:45.338 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.325+0000 7f6d795cf640 1 -- 172.21.15.98:0/3503441533 wait complete. 2024-03-29T11:58:45.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.326+0000 7f6d795cf640 10 monclient: build_initial_monmap 2024-03-29T11:58:45.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.326+0000 7f6d795cf640 10 monclient: monmap: 2024-03-29T11:58:45.342 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:58:45.342 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:58:45.342 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:58:45.342 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:58:45.342 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:58:45.342 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:45.342 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:58:45.343 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:58:45.343 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:58:45.343 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:45.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.327+0000 7f6d795cf640 1 librados: starting msgr at 2024-03-29T11:58:45.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.327+0000 7f6d795cf640 1 librados: starting objecter 2024-03-29T11:58:45.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.327+0000 7f6d795cf640 1 Processor -- start 2024-03-29T11:58:45.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.327+0000 7f6d795cf640 1 -- start start 2024-03-29T11:58:45.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.327+0000 7f6d795cf640 1 librados: setting wanted keys 2024-03-29T11:58:45.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.327+0000 7f6d795cf640 1 librados: calling monclient init 2024-03-29T11:58:45.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.327+0000 7f6d795cf640 10 monclient: init 2024-03-29T11:58:45.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.328+0000 7f6d795cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:58:45.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.328+0000 7f6d795cf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:58:45.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.328+0000 7f6d795cf640 10 monclient(hunting): picked mon.noname-b con 0x7f6d740b4890 addr v1:172.21.15.121:6789/0 2024-03-29T11:58:45.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.328+0000 7f6d795cf640 10 monclient(hunting): picked mon.noname-c con 0x7f6d7413c040 addr v1:172.21.15.98:6790/0 2024-03-29T11:58:45.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.328+0000 7f6d795cf640 10 monclient(hunting): picked mon.noname-a con 0x7f6d741427e0 addr v1:172.21.15.98:6789/0 2024-03-29T11:58:45.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.328+0000 7f6d795cf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d74070180 con 0x7f6d741427e0 2024-03-29T11:58:45.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.328+0000 7f6d795cf640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d740712e0 con 0x7f6d740b4890 2024-03-29T11:58:45.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.328+0000 7f6d737fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f6d741427e0 0x7f6d7413a0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38140/0 (socket says 172.21.15.98:38140) 2024-03-29T11:58:45.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.328+0000 7f6d737fe640 1 -- 172.21.15.98:0/2671987163 learned_addr learned my addr 172.21.15.98:0/2671987163 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:58:45.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.328+0000 7f6d795cf640 1 -- 172.21.15.98:0/2671987163 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d740715c0 con 0x7f6d7413c040 2024-03-29T11:58:45.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.328+0000 7f6d5b7fe640 10 client.?.objecter ms_handle_connect 0x7f6d741427e0 2024-03-29T11:58:45.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.328+0000 7f6d795cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:58:45.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.328+0000 7f6d795cf640 10 monclient(hunting): authenticate will time out at 1432.413828s 2024-03-29T11:58:45.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.328+0000 7f6d5b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:58:45.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.328+0000 7f6d5b7fe640 10 client.?.objecter ms_handle_connect 0x7f6d7413c040 2024-03-29T11:58:45.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.328+0000 7f6d5b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:58:45.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.329+0000 7f6d5b7fe640 1 -- 172.21.15.98:0/2671987163 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2116587379 0 0) 0x7f6d740715c0 con 0x7f6d7413c040 2024-03-29T11:58:45.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.329+0000 7f6d5b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:58:45.344 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.329+0000 7f6d5b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:58:45.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.329+0000 7f6d5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:58:45.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.329+0000 7f6d5b7fe640 10 monclient(hunting): my global_id is 4634 2024-03-29T11:58:45.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.329+0000 7f6d5b7fe640 1 -- 172.21.15.98:0/2671987163 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d60004eb0 con 0x7f6d7413c040 2024-03-29T11:58:45.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.329+0000 7f6d5b7fe640 10 client.?.objecter ms_handle_connect 0x7f6d740b4890 2024-03-29T11:58:45.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.329+0000 7f6d5b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:58:45.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.329+0000 7f6d5b7fe640 1 -- 172.21.15.98:0/2671987163 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2983720141 0 0) 0x7f6d74070180 con 0x7f6d741427e0 2024-03-29T11:58:45.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.329+0000 7f6d5b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:58:45.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.329+0000 7f6d5b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:58:45.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.329+0000 7f6d5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:58:45.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.329+0000 7f6d5b7fe640 10 monclient(hunting): my global_id is 4587 2024-03-29T11:58:45.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.329+0000 7f6d5b7fe640 1 -- 172.21.15.98:0/2671987163 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d740715c0 con 0x7f6d741427e0 2024-03-29T11:58:45.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.329+0000 7f6d5b7fe640 1 -- 172.21.15.98:0/2671987163 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2111243857 0 0) 0x7f6d60004eb0 con 0x7f6d7413c040 2024-03-29T11:58:45.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.329+0000 7f6d5b7fe640 1 -- 172.21.15.98:0/2671987163 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d74070180 con 0x7f6d7413c040 2024-03-29T11:58:45.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.329+0000 7f6d5b7fe640 1 -- 172.21.15.98:0/2671987163 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f6d680018c0 con 0x7f6d7413c040 2024-03-29T11:58:45.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.329+0000 7f6d5b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:58:45.345 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.329+0000 7f6d5b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:58:45.346 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.329+0000 7f6d5b7fe640 10 monclient(hunting): dump: 2024-03-29T11:58:45.346 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:58:45.346 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:45.346 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:45.346 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:45.346 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:58:45.346 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:45.346 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:58:45.346 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:58:45.346 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:58:45.346 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:45.346 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.329+0000 7f6d5b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:58:45.346 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.329+0000 7f6d5b7fe640 1 -- 172.21.15.98:0/2671987163 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2751409207 0 0) 0x7f6d740712e0 con 0x7f6d740b4890 2024-03-29T11:58:45.346 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.329+0000 7f6d5b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:58:45.346 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.329+0000 7f6d5b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:58:45.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.329+0000 7f6d5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:58:45.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.329+0000 7f6d5b7fe640 10 monclient(hunting): my global_id is 4579 2024-03-29T11:58:45.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.329+0000 7f6d5b7fe640 1 -- 172.21.15.98:0/2671987163 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d60004eb0 con 0x7f6d740b4890 2024-03-29T11:58:45.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.330+0000 7f6d5b7fe640 1 -- 172.21.15.98:0/2671987163 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 246532925 0 0) 0x7f6d740715c0 con 0x7f6d741427e0 2024-03-29T11:58:45.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.330+0000 7f6d5b7fe640 1 -- 172.21.15.98:0/2671987163 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d740712e0 con 0x7f6d741427e0 2024-03-29T11:58:45.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.330+0000 7f6d5b7fe640 1 -- 172.21.15.98:0/2671987163 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f6d6c001f00 con 0x7f6d741427e0 2024-03-29T11:58:45.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.330+0000 7f6d5b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:58:45.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.330+0000 7f6d5b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:58:45.347 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.330+0000 7f6d5b7fe640 10 monclient(hunting): dump: 2024-03-29T11:58:45.347 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:58:45.347 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:45.347 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:45.348 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:45.348 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:58:45.348 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:45.348 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:58:45.348 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:58:45.348 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:58:45.348 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:45.348 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.330+0000 7f6d5b7fe640 1 -- 172.21.15.98:0/2671987163 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2264858899 0 0) 0x7f6d740712e0 con 0x7f6d741427e0 2024-03-29T11:58:45.348 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.330+0000 7f6d5b7fe640 1 -- 172.21.15.98:0/2671987163 >> v1:172.21.15.98:6790/0 conn(0x7f6d7413c040 legacy=0x7f6d7406de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:45.348 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.330+0000 7f6d5b7fe640 1 -- 172.21.15.98:0/2671987163 >> v1:172.21.15.121:6789/0 conn(0x7f6d740b4890 legacy=0x7f6d7413e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:45.348 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.330+0000 7f6d5b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:58:45.348 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.330+0000 7f6d5b7fe640 1 monclient: found mon.a 2024-03-29T11:58:45.348 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.330+0000 7f6d5b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:58:45.348 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.330+0000 7f6d5b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:58:45.348 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.330+0000 7f6d5b7fe640 1 -- 172.21.15.98:0/2671987163 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d740718a0 con 0x7f6d741427e0 2024-03-29T11:58:45.349 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.330+0000 7f6d5b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:58:45.349 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.330+0000 7f6d5b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:58:45.349 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.330+0000 7f6d5b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:58:45.349 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.330+0000 7f6d795cf640 5 monclient: authenticate success, global_id 4587 2024-03-29T11:58:45.349 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.330+0000 7f6d795cf640 10 monclient: _renew_subs 2024-03-29T11:58:45.349 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.330+0000 7f6d795cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:58:45.349 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.330+0000 7f6d795cf640 1 -- 172.21.15.98:0/2671987163 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d7413a7b0 con 0x7f6d741427e0 2024-03-29T11:58:45.349 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.330+0000 7f6d795cf640 10 client.4587.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:58:45.349 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.330+0000 7f6d795cf640 10 monclient: _renew_subs 2024-03-29T11:58:45.349 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.330+0000 7f6d795cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:58:45.349 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.330+0000 7f6d795cf640 1 -- 172.21.15.98:0/2671987163 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d740b14c0 con 0x7f6d741427e0 2024-03-29T11:58:45.349 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.330+0000 7f6d795cf640 1 librados: init done 2024-03-29T11:58:45.349 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.330+0000 7f6d5b7fe640 1 -- 172.21.15.98:0/2671987163 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d6c001570 con 0x7f6d741427e0 2024-03-29T11:58:45.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.330+0000 7f6d5b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:58:45.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.331+0000 7f6d5b7fe640 1 -- 172.21.15.98:0/2671987163 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f6d6c004fb0 con 0x7f6d741427e0 2024-03-29T11:58:45.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.331+0000 7f6d5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:58:45.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.331+0000 7f6d5b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:58:45.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.331+0000 7f6d5b7fe640 10 monclient: dump: 2024-03-29T11:58:45.350 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:58:45.350 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:45.350 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:45.350 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:45.350 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:58:45.350 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:45.350 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:58:45.350 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:58:45.350 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:58:45.351 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:45.351 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.331+0000 7f6d717fa640 4 set_mon_vals no callback set 2024-03-29T11:58:45.351 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.332+0000 7f6d795cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:58:45.351 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.332+0000 7f6d795cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:58:45.351 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.332+0000 7f6d795cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:58:45.351 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.332+0000 7f6d795cf640 1 -- 172.21.15.98:0/2671987163 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d54005080 con 0x7f6d741427e0 2024-03-29T11:58:45.351 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.333+0000 7f6d5b7fe640 1 -- 172.21.15.98:0/2671987163 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f6d6c005110 con 0x7f6d741427e0 2024-03-29T11:58:45.351 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.334+0000 7f6d5b7fe640 1 -- 172.21.15.98:0/2671987163 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3033626086 0 0) 0x7f6d6c084f40 con 0x7f6d741427e0 2024-03-29T11:58:45.351 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.334+0000 7f6d72ffd640 10 monclient: get_auth_request con 0x7f6d6006df00 auth_method 0 2024-03-29T11:58:45.351 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.334+0000 7f6d5b7fe640 10 client.4587.objecter ms_dispatch 0x7f6d74000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:58:45.351 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.334+0000 7f6d5b7fe640 3 client.4587.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:58:45.351 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.334+0000 7f6d5b7fe640 3 client.4587.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:58:45.351 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.334+0000 7f6d5b7fe640 20 client.4587.objecter dump_active .. 0 homeless 2024-03-29T11:58:45.351 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.334+0000 7f6d5b7fe640 10 client.4587.objecter ms_handle_connect 0x7f6d6006df00 2024-03-29T11:58:45.352 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.341+0000 7f6d5b7fe640 1 -- 172.21.15.98:0/2671987163 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6d6c050880 con 0x7f6d741427e0 2024-03-29T11:58:45.352 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.341+0000 7f6d5b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:58:45.352 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.341+0000 7f6d5b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:58:45.724 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.717+0000 7f6d795cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:58:45.724 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.717+0000 7f6d795cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:58:45.724 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.717+0000 7f6d795cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:58:45.724 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.717+0000 7f6d795cf640 1 -- 172.21.15.98:0/2671987163 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d54005330 con 0x7f6d741427e0 2024-03-29T11:58:45.725 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.718+0000 7f6d5b7fe640 1 -- 172.21.15.98:0/2671987163 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3141891755) 0x7f6d6c0845f0 con 0x7f6d741427e0 2024-03-29T11:58:45.725 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.718+0000 7f6d5b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:58:45.725 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.718+0000 7f6d5b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:58:45.725 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:58:45.725 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":11,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:58:22.774801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:58:22.774801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:58:45.728 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.722+0000 7f6d795cf640 10 librados: watch_flush enter 2024-03-29T11:58:45.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.722+0000 7f6d795cf640 10 librados: watch_flush exit 2024-03-29T11:58:45.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.722+0000 7f6d795cf640 20 client.4587.objecter shutdown clearing up homeless session... 2024-03-29T11:58:45.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.722+0000 7f6d795cf640 10 client.4587.objecter successfully canceled tick 2024-03-29T11:58:45.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.722+0000 7f6d795cf640 1 -- 172.21.15.98:0/2671987163 >> v1:172.21.15.98:6816/4065186974 conn(0x7f6d6006df00 legacy=0x7f6d600703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:45.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.722+0000 7f6d795cf640 10 monclient: shutdown 2024-03-29T11:58:45.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.722+0000 7f6d795cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:58:45.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.722+0000 7f6d795cf640 1 -- 172.21.15.98:0/2671987163 >> v1:172.21.15.98:6789/0 conn(0x7f6d741427e0 legacy=0x7f6d7413a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:45.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.722+0000 7f6d795cf640 1 -- 172.21.15.98:0/2671987163 shutdown_connections 2024-03-29T11:58:45.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.722+0000 7f6d795cf640 1 -- 172.21.15.98:0/2671987163 >> 172.21.15.98:0/2671987163 conn(0x7f6d7406d180 msgr2=0x7f6d740b5b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:58:45.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.722+0000 7f6d795cf640 1 -- 172.21.15.98:0/2671987163 shutdown_connections 2024-03-29T11:58:45.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.722+0000 7f6d795cf640 1 -- 172.21.15.98:0/2671987163 wait complete. 2024-03-29T11:58:45.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:45.722+0000 7f6d795cf640 1 librados: shutdown 2024-03-29T11:58:45.747 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:58:52.554 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:58:52.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.701+0000 7f2dfbd5d640 10 monclient: get_monmap_and_config 2024-03-29T11:58:52.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.701+0000 7f2dfbd5d640 10 monclient: build_initial_monmap 2024-03-29T11:58:52.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.701+0000 7f2dfbd5d640 10 monclient: monmap: 2024-03-29T11:58:52.708 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:58:52.708 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:52.708 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:58:52.702454+0000 2024-03-29T11:58:52.709 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:58:52.702454+0000 2024-03-29T11:58:52.709 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:58:52.709 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:52.709 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:58:52.709 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:58:52.709 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:58:52.709 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:52.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.702+0000 7f2dfbd5d640 1 Processor -- start 2024-03-29T11:58:52.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.702+0000 7f2dfbd5d640 1 -- start start 2024-03-29T11:58:52.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.702+0000 7f2dfbd5d640 10 monclient: init 2024-03-29T11:58:52.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.703+0000 7f2dfbd5d640 10 monclient: _reopen_session rank -1 2024-03-29T11:58:52.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.703+0000 7f2dfbd5d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:58:52.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.703+0000 7f2dfbd5d640 10 monclient(hunting): picked mon.noname-b con 0x7f2df41422b0 addr v1:172.21.15.98:6790/0 2024-03-29T11:58:52.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.703+0000 7f2dfbd5d640 10 monclient(hunting): picked mon.noname-c con 0x7f2df4142690 addr v1:172.21.15.121:6789/0 2024-03-29T11:58:52.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.704+0000 7f2dfbd5d640 10 monclient(hunting): picked mon.noname-a con 0x7f2df40b17d0 addr v1:172.21.15.98:6789/0 2024-03-29T11:58:52.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.704+0000 7f2df92f8640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f2df4142690 0x7f2df40b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:40350/0 (socket says 172.21.15.98:40350) 2024-03-29T11:58:52.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.704+0000 7f2df92f8640 1 -- 172.21.15.98:0/1135649527 learned_addr learned my addr 172.21.15.98:0/1135649527 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:58:52.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.704+0000 7f2df9af9640 1 --1- 172.21.15.98:0/1135649527 >> v1:172.21.15.98:6790/0 conn(0x7f2df41422b0 0x7f2df413f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:60192/0 (socket says 172.21.15.98:60192) 2024-03-29T11:58:52.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.704+0000 7f2dfbd5d640 1 -- 172.21.15.98:0/1135649527 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2df4142c80 con 0x7f2df40b17d0 2024-03-29T11:58:52.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.704+0000 7f2dfbd5d640 1 -- 172.21.15.98:0/1135649527 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2df40aae40 con 0x7f2df4142690 2024-03-29T11:58:52.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.704+0000 7f2dfbd5d640 1 -- 172.21.15.98:0/1135649527 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2df413fbd0 con 0x7f2df41422b0 2024-03-29T11:58:52.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.704+0000 7f2dfbd5d640 10 monclient(hunting): _renew_subs 2024-03-29T11:58:52.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.704+0000 7f2dfbd5d640 10 monclient(hunting): authenticate will time out at 1439.789490s 2024-03-29T11:58:52.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.704+0000 7f2df8af7640 1 -- 172.21.15.98:0/1135649527 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3934590125 0 0) 0x7f2df40aae40 con 0x7f2df4142690 2024-03-29T11:58:52.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.704+0000 7f2df8af7640 10 monclient(hunting): _negotiate 2024-03-29T11:58:52.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.704+0000 7f2df8af7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:58:52.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.704+0000 7f2df8af7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:58:52.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.704+0000 7f2df8af7640 10 monclient(hunting): my global_id is 4582 2024-03-29T11:58:52.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.704+0000 7f2df8af7640 1 -- 172.21.15.98:0/1135649527 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2de0003520 con 0x7f2df4142690 2024-03-29T11:58:52.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.704+0000 7f2df8af7640 1 -- 172.21.15.98:0/1135649527 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1823138036 0 0) 0x7f2df413fbd0 con 0x7f2df41422b0 2024-03-29T11:58:52.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.704+0000 7f2df8af7640 10 monclient(hunting): _negotiate 2024-03-29T11:58:52.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.704+0000 7f2df8af7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:58:52.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.704+0000 7f2df8af7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:58:52.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.704+0000 7f2df8af7640 10 monclient(hunting): my global_id is 4637 2024-03-29T11:58:52.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.704+0000 7f2df8af7640 1 -- 172.21.15.98:0/1135649527 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2df40aae40 con 0x7f2df41422b0 2024-03-29T11:58:52.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.704+0000 7f2df8af7640 1 -- 172.21.15.98:0/1135649527 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3686702922 0 0) 0x7f2df4142c80 con 0x7f2df40b17d0 2024-03-29T11:58:52.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.704+0000 7f2df8af7640 10 monclient(hunting): _negotiate 2024-03-29T11:58:52.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.704+0000 7f2df8af7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:58:52.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.704+0000 7f2df8af7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:58:52.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.704+0000 7f2df8af7640 10 monclient(hunting): my global_id is 4590 2024-03-29T11:58:52.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.704+0000 7f2df8af7640 1 -- 172.21.15.98:0/1135649527 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2df413fbd0 con 0x7f2df40b17d0 2024-03-29T11:58:52.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.705+0000 7f2df8af7640 1 -- 172.21.15.98:0/1135649527 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1822064978 0 0) 0x7f2df40aae40 con 0x7f2df41422b0 2024-03-29T11:58:52.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.705+0000 7f2df8af7640 1 -- 172.21.15.98:0/1135649527 >> v1:172.21.15.121:6789/0 conn(0x7f2df4142690 legacy=0x7f2df40b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:52.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.705+0000 7f2df8af7640 1 -- 172.21.15.98:0/1135649527 >> v1:172.21.15.98:6789/0 conn(0x7f2df40b17d0 legacy=0x7f2df40b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:52.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.705+0000 7f2df8af7640 10 monclient: _finish_hunting 0 2024-03-29T11:58:52.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.705+0000 7f2df8af7640 1 monclient: found mon.noname-b 2024-03-29T11:58:52.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.705+0000 7f2df8af7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:58:52.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.705+0000 7f2df8af7640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:58:52.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.705+0000 7f2df8af7640 1 -- 172.21.15.98:0/1135649527 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df4151940 con 0x7f2df41422b0 2024-03-29T11:58:52.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.705+0000 7f2df8af7640 10 monclient: _finish_auth 0 2024-03-29T11:58:52.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.705+0000 7f2df8af7640 10 monclient: _check_auth_tickets 2024-03-29T11:58:52.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.705+0000 7f2df8af7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:58:52.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.705+0000 7f2df8af7640 1 -- 172.21.15.98:0/1135649527 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f2df0001670 con 0x7f2df41422b0 2024-03-29T11:58:52.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.705+0000 7f2df8af7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:58:52.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.705+0000 7f2df8af7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:58:52.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.705+0000 7f2df8af7640 10 monclient: dump: 2024-03-29T11:58:52.715 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:58:52.715 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:52.715 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:52.715 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:52.715 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:58:52.715 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:52.715 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:58:52.715 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:58:52.715 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:58:52.715 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:52.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.705+0000 7f2dfbd5d640 5 monclient: authenticate success, global_id 4637 2024-03-29T11:58:52.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.705+0000 7f2dfbd5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:58:52.724 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.705+0000 7f2df8af7640 1 -- 172.21.15.98:0/1135649527 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2df00017d0 con 0x7f2df41422b0 2024-03-29T11:58:52.724 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.705+0000 7f2df8af7640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:58:52.724 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.705+0000 7f2dfbd5d640 10 monclient: get_monmap_and_config success 2024-03-29T11:58:52.724 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.705+0000 7f2df8af7640 1 -- 172.21.15.98:0/1135649527 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f2df0001ba0 con 0x7f2df41422b0 2024-03-29T11:58:52.725 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.705+0000 7f2df8af7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:58:52.725 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.705+0000 7f2dfbd5d640 4 set_mon_vals no callback set 2024-03-29T11:58:52.725 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.705+0000 7f2df8af7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:58:52.725 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.705+0000 7f2df8af7640 10 monclient: dump: 2024-03-29T11:58:52.725 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:58:52.725 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:52.725 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:52.725 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:52.725 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:58:52.725 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:52.725 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:58:52.725 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:58:52.725 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:58:52.725 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:52.725 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.706+0000 7f2dfbd5d640 10 monclient: shutdown 2024-03-29T11:58:52.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.706+0000 7f2dfbd5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:58:52.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.706+0000 7f2dfbd5d640 1 -- 172.21.15.98:0/1135649527 >> v1:172.21.15.98:6790/0 conn(0x7f2df41422b0 legacy=0x7f2df413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:52.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.706+0000 7f2dfbd5d640 1 -- 172.21.15.98:0/1135649527 shutdown_connections 2024-03-29T11:58:52.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.706+0000 7f2dfbd5d640 1 -- 172.21.15.98:0/1135649527 >> 172.21.15.98:0/1135649527 conn(0x7f2df4066a00 msgr2=0x7f2df4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:58:52.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.706+0000 7f2dfbd5d640 1 -- 172.21.15.98:0/1135649527 shutdown_connections 2024-03-29T11:58:52.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.706+0000 7f2dfbd5d640 1 -- 172.21.15.98:0/1135649527 wait complete. 2024-03-29T11:58:52.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.707+0000 7f2dfbd5d640 10 monclient: build_initial_monmap 2024-03-29T11:58:52.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.707+0000 7f2dfbd5d640 10 monclient: monmap: 2024-03-29T11:58:52.726 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:58:52.726 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:58:52.726 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:58:52.726 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:58:52.726 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:58:52.726 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:52.727 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:58:52.727 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:58:52.727 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:58:52.727 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:52.727 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.707+0000 7f2dfbd5d640 1 librados: starting msgr at 2024-03-29T11:58:52.727 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.707+0000 7f2dfbd5d640 1 librados: starting objecter 2024-03-29T11:58:52.727 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.708+0000 7f2dfbd5d640 1 Processor -- start 2024-03-29T11:58:52.727 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.708+0000 7f2dfbd5d640 1 -- start start 2024-03-29T11:58:52.727 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.708+0000 7f2dfbd5d640 1 librados: setting wanted keys 2024-03-29T11:58:52.727 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.708+0000 7f2dfbd5d640 1 librados: calling monclient init 2024-03-29T11:58:52.727 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.708+0000 7f2dfbd5d640 10 monclient: init 2024-03-29T11:58:52.727 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.709+0000 7f2dfbd5d640 10 monclient: _reopen_session rank -1 2024-03-29T11:58:52.728 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.709+0000 7f2dfbd5d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:58:52.728 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.709+0000 7f2dfbd5d640 10 monclient(hunting): picked mon.noname-c con 0x7f2df40b17d0 addr v1:172.21.15.98:6790/0 2024-03-29T11:58:52.728 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.709+0000 7f2dfbd5d640 10 monclient(hunting): picked mon.noname-a con 0x7f2df41422b0 addr v1:172.21.15.98:6789/0 2024-03-29T11:58:52.728 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.709+0000 7f2dfbd5d640 10 monclient(hunting): picked mon.noname-b con 0x7f2df4142690 addr v1:172.21.15.121:6789/0 2024-03-29T11:58:52.728 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.709+0000 7f2dfbd5d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2df412f680 con 0x7f2df41422b0 2024-03-29T11:58:52.728 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.709+0000 7f2dfbd5d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2df40b7530 con 0x7f2df4142690 2024-03-29T11:58:52.728 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.709+0000 7f2df9af9640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f2df40b17d0 0x7f2df41dacf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:60596/0 (socket says 172.21.15.98:60596) 2024-03-29T11:58:52.728 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.709+0000 7f2df9af9640 1 -- 172.21.15.98:0/3087681072 learned_addr learned my addr 172.21.15.98:0/3087681072 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:58:52.728 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.709+0000 7f2dfbd5d640 1 -- 172.21.15.98:0/3087681072 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2df40acbd0 con 0x7f2df40b17d0 2024-03-29T11:58:52.728 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.709+0000 7f2dfbd5d640 10 monclient(hunting): _renew_subs 2024-03-29T11:58:52.728 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.709+0000 7f2dfbd5d640 10 monclient(hunting): authenticate will time out at 1439.794806s 2024-03-29T11:58:52.728 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.709+0000 7f2de67fc640 10 client.?.objecter ms_handle_connect 0x7f2df40b17d0 2024-03-29T11:58:52.728 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.709+0000 7f2de67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:58:52.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.710+0000 7f2de67fc640 10 client.?.objecter ms_handle_connect 0x7f2df41422b0 2024-03-29T11:58:52.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.710+0000 7f2de67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:58:52.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.710+0000 7f2de67fc640 10 client.?.objecter ms_handle_connect 0x7f2df4142690 2024-03-29T11:58:52.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.710+0000 7f2de67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:58:52.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.710+0000 7f2de67fc640 1 -- 172.21.15.98:0/3087681072 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1101642188 0 0) 0x7f2df40acbd0 con 0x7f2df40b17d0 2024-03-29T11:58:52.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.710+0000 7f2de67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:58:52.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.710+0000 7f2de67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:58:52.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.710+0000 7f2de67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:58:52.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.710+0000 7f2de67fc640 10 monclient(hunting): my global_id is 4640 2024-03-29T11:58:52.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.710+0000 7f2de67fc640 1 -- 172.21.15.98:0/3087681072 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dd8004eb0 con 0x7f2df40b17d0 2024-03-29T11:58:52.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.710+0000 7f2de67fc640 1 -- 172.21.15.98:0/3087681072 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3626803743 0 0) 0x7f2df40b7530 con 0x7f2df4142690 2024-03-29T11:58:52.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.710+0000 7f2de67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:58:52.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.710+0000 7f2de67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:58:52.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.710+0000 7f2de67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:58:52.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.710+0000 7f2de67fc640 10 monclient(hunting): my global_id is 4585 2024-03-29T11:58:52.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.710+0000 7f2de67fc640 1 -- 172.21.15.98:0/3087681072 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2df40acbd0 con 0x7f2df4142690 2024-03-29T11:58:52.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.710+0000 7f2de67fc640 1 -- 172.21.15.98:0/3087681072 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 512387148 0 0) 0x7f2df412f680 con 0x7f2df41422b0 2024-03-29T11:58:52.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.710+0000 7f2de67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:58:52.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.710+0000 7f2de67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:58:52.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.710+0000 7f2de67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:58:52.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.710+0000 7f2de67fc640 10 monclient(hunting): my global_id is 4593 2024-03-29T11:58:52.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.710+0000 7f2de67fc640 1 -- 172.21.15.98:0/3087681072 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2df40b7530 con 0x7f2df41422b0 2024-03-29T11:58:52.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.710+0000 7f2de67fc640 1 -- 172.21.15.98:0/3087681072 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3237581189 0 0) 0x7f2df40acbd0 con 0x7f2df4142690 2024-03-29T11:58:52.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.710+0000 7f2de67fc640 1 -- 172.21.15.98:0/3087681072 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2df412f680 con 0x7f2df4142690 2024-03-29T11:58:52.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.710+0000 7f2de67fc640 1 -- 172.21.15.98:0/3087681072 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f2dec001bf0 con 0x7f2df4142690 2024-03-29T11:58:52.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.710+0000 7f2de67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:58:52.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.710+0000 7f2de67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:58:52.731 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.710+0000 7f2de67fc640 10 monclient(hunting): dump: 2024-03-29T11:58:52.731 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:58:52.731 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:52.731 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:52.731 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:52.731 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:58:52.731 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:52.731 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:58:52.731 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:58:52.731 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:58:52.731 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:52.731 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.710+0000 7f2de67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:58:52.731 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.710+0000 7f2de67fc640 1 -- 172.21.15.98:0/3087681072 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2394346529 0 0) 0x7f2dd8004eb0 con 0x7f2df40b17d0 2024-03-29T11:58:52.731 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.710+0000 7f2de67fc640 1 -- 172.21.15.98:0/3087681072 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2df40acbd0 con 0x7f2df40b17d0 2024-03-29T11:58:52.732 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.711+0000 7f2de67fc640 1 -- 172.21.15.98:0/3087681072 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f2df0001b60 con 0x7f2df40b17d0 2024-03-29T11:58:52.732 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.711+0000 7f2de67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:58:52.732 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.711+0000 7f2de67fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:58:52.732 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.711+0000 7f2de67fc640 10 monclient(hunting): dump: 2024-03-29T11:58:52.732 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:58:52.732 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:52.732 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:52.732 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:52.732 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:58:52.732 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:52.732 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:58:52.732 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:58:52.732 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:58:52.733 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:52.733 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.711+0000 7f2de67fc640 1 -- 172.21.15.98:0/3087681072 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 863916655 0 0) 0x7f2df412f680 con 0x7f2df4142690 2024-03-29T11:58:52.733 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.711+0000 7f2de67fc640 1 -- 172.21.15.98:0/3087681072 >> v1:172.21.15.98:6790/0 conn(0x7f2df40b17d0 legacy=0x7f2df41dacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:52.733 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.711+0000 7f2de67fc640 1 -- 172.21.15.98:0/3087681072 >> v1:172.21.15.98:6789/0 conn(0x7f2df41422b0 legacy=0x7f2df412e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:52.733 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.711+0000 7f2de67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:58:52.733 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.711+0000 7f2de67fc640 1 monclient: found mon.b 2024-03-29T11:58:52.733 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.711+0000 7f2de67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:58:52.733 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.711+0000 7f2de67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:58:52.733 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.711+0000 7f2de67fc640 1 -- 172.21.15.98:0/3087681072 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df40aceb0 con 0x7f2df4142690 2024-03-29T11:58:52.733 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.711+0000 7f2de67fc640 10 monclient: _finish_auth 0 2024-03-29T11:58:52.733 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.711+0000 7f2de67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:58:52.733 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.711+0000 7f2de67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:58:52.734 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.711+0000 7f2dfbd5d640 5 monclient: authenticate success, global_id 4585 2024-03-29T11:58:52.734 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.711+0000 7f2dfbd5d640 10 monclient: _renew_subs 2024-03-29T11:58:52.734 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.711+0000 7f2dfbd5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:58:52.734 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.711+0000 7f2dfbd5d640 1 -- 172.21.15.98:0/3087681072 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2df40ad190 con 0x7f2df4142690 2024-03-29T11:58:52.734 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.711+0000 7f2dfbd5d640 10 client.4585.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:58:52.734 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.711+0000 7f2dfbd5d640 10 monclient: _renew_subs 2024-03-29T11:58:52.734 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.711+0000 7f2dfbd5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:58:52.734 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.711+0000 7f2dfbd5d640 1 -- 172.21.15.98:0/3087681072 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2df40ab840 con 0x7f2df4142690 2024-03-29T11:58:52.734 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.711+0000 7f2dfbd5d640 1 librados: init done 2024-03-29T11:58:52.734 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.711+0000 7f2de67fc640 1 -- 172.21.15.98:0/3087681072 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2dec0020c0 con 0x7f2df4142690 2024-03-29T11:58:52.734 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.711+0000 7f2de67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:58:52.734 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.711+0000 7f2de67fc640 1 -- 172.21.15.98:0/3087681072 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f2dec003b30 con 0x7f2df4142690 2024-03-29T11:58:52.734 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.711+0000 7f2de67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:58:52.735 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.711+0000 7f2de67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:58:52.735 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.711+0000 7f2de67fc640 10 monclient: dump: 2024-03-29T11:58:52.735 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:58:52.735 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:58:52.735 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:52.735 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:58:52.735 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:58:52.735 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:58:52.735 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:58:52.735 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:58:52.735 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:58:52.735 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:58:52.735 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.711+0000 7f2de7fff640 4 set_mon_vals no callback set 2024-03-29T11:58:52.736 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.712+0000 7f2dfbd5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:58:52.736 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.712+0000 7f2dfbd5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:58:52.736 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.712+0000 7f2dfbd5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:58:52.736 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.713+0000 7f2dfbd5d640 1 -- 172.21.15.98:0/3087681072 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ddc005080 con 0x7f2df4142690 2024-03-29T11:58:52.736 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.713+0000 7f2de67fc640 1 -- 172.21.15.98:0/3087681072 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f2dec003c90 con 0x7f2df4142690 2024-03-29T11:58:52.736 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.714+0000 7f2de67fc640 1 -- 172.21.15.98:0/3087681072 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3033626086 0 0) 0x7f2dec083a50 con 0x7f2df4142690 2024-03-29T11:58:52.736 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.714+0000 7f2df9af9640 10 monclient: get_auth_request con 0x7f2dd806de30 auth_method 0 2024-03-29T11:58:52.736 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.714+0000 7f2de67fc640 10 client.4585.objecter ms_dispatch 0x7f2df4000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:58:52.736 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.714+0000 7f2de67fc640 3 client.4585.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:58:52.736 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.714+0000 7f2de67fc640 3 client.4585.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:58:52.736 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.714+0000 7f2de67fc640 20 client.4585.objecter dump_active .. 0 homeless 2024-03-29T11:58:52.736 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.714+0000 7f2de67fc640 10 client.4585.objecter ms_handle_connect 0x7f2dd806de30 2024-03-29T11:58:52.737 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.721+0000 7f2de67fc640 1 -- 172.21.15.98:0/3087681072 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2dec04f330 con 0x7f2df4142690 2024-03-29T11:58:52.737 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.721+0000 7f2de67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:58:52.737 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:52.721+0000 7f2de67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:58:53.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:53.062+0000 7f2dfbd5d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:58:53.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:53.062+0000 7f2dfbd5d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:58:53.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:53.062+0000 7f2dfbd5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:58:53.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:53.062+0000 7f2dfbd5d640 1 -- 172.21.15.98:0/3087681072 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ddc005330 con 0x7f2df4142690 2024-03-29T11:58:53.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:53.063+0000 7f2de67fc640 1 -- 172.21.15.98:0/3087681072 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3141891755) 0x7f2dec051260 con 0x7f2df4142690 2024-03-29T11:58:53.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:53.063+0000 7f2de67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:58:53.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:53.063+0000 7f2de67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:58:53.071 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:58:53.071 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":11,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:58:22.774801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:58:22.774801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:58:53.074 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:53.067+0000 7f2dfbd5d640 10 librados: watch_flush enter 2024-03-29T11:58:53.074 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:53.067+0000 7f2dfbd5d640 10 librados: watch_flush exit 2024-03-29T11:58:53.074 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:53.067+0000 7f2dfbd5d640 20 client.4585.objecter shutdown clearing up homeless session... 2024-03-29T11:58:53.074 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:53.067+0000 7f2dfbd5d640 10 client.4585.objecter successfully canceled tick 2024-03-29T11:58:53.074 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:53.067+0000 7f2dfbd5d640 1 -- 172.21.15.98:0/3087681072 >> v1:172.21.15.98:6816/4065186974 conn(0x7f2dd806de30 legacy=0x7f2dd80702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:53.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:53.067+0000 7f2dfbd5d640 10 monclient: shutdown 2024-03-29T11:58:53.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:53.067+0000 7f2dfbd5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:58:53.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:53.067+0000 7f2dfbd5d640 1 -- 172.21.15.98:0/3087681072 >> v1:172.21.15.121:6789/0 conn(0x7f2df4142690 legacy=0x7f2df412ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:58:53.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:53.067+0000 7f2dfbd5d640 1 -- 172.21.15.98:0/3087681072 shutdown_connections 2024-03-29T11:58:53.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:53.067+0000 7f2dfbd5d640 1 -- 172.21.15.98:0/3087681072 >> 172.21.15.98:0/3087681072 conn(0x7f2df4066a00 msgr2=0x7f2df4131e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:58:53.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:53.068+0000 7f2dfbd5d640 1 -- 172.21.15.98:0/3087681072 shutdown_connections 2024-03-29T11:58:53.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:53.068+0000 7f2dfbd5d640 1 -- 172.21.15.98:0/3087681072 wait complete. 2024-03-29T11:58:53.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:58:53.068+0000 7f2dfbd5d640 1 librados: shutdown 2024-03-29T11:58:53.089 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:58:59.896 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:59:00.046 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.038+0000 7f73fe8cf640 10 monclient: get_monmap_and_config 2024-03-29T11:59:00.046 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.038+0000 7f73fe8cf640 10 monclient: build_initial_monmap 2024-03-29T11:59:00.046 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.038+0000 7f73fe8cf640 10 monclient: monmap: 2024-03-29T11:59:00.046 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:59:00.046 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:59:00.047 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:59:00.039548+0000 2024-03-29T11:59:00.047 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:59:00.039548+0000 2024-03-29T11:59:00.047 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:59:00.047 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:59:00.047 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:59:00.047 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:59:00.047 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:59:00.047 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:59:00.047 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.039+0000 7f73fe8cf640 1 Processor -- start 2024-03-29T11:59:00.047 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.039+0000 7f73fe8cf640 1 -- start start 2024-03-29T11:59:00.047 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.039+0000 7f73fe8cf640 10 monclient: init 2024-03-29T11:59:00.050 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.040+0000 7f73fe8cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:59:00.051 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.040+0000 7f73fe8cf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:59:00.051 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.040+0000 7f73fe8cf640 10 monclient(hunting): picked mon.noname-a con 0x7f73f80a96c0 addr v1:172.21.15.98:6789/0 2024-03-29T11:59:00.051 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.040+0000 7f73fe8cf640 10 monclient(hunting): picked mon.noname-b con 0x7f73f80a9aa0 addr v1:172.21.15.98:6790/0 2024-03-29T11:59:00.051 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.040+0000 7f73fe8cf640 10 monclient(hunting): picked mon.noname-c con 0x7f73f80ad730 addr v1:172.21.15.121:6789/0 2024-03-29T11:59:00.051 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.040+0000 7f73fe8cf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73f813fee0 con 0x7f73f80a96c0 2024-03-29T11:59:00.051 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.040+0000 7f73fe8cf640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73f80aab80 con 0x7f73f80ad730 2024-03-29T11:59:00.051 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.040+0000 7f73fe8cf640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73f8145e80 con 0x7f73f80a9aa0 2024-03-29T11:59:00.051 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.040+0000 7f73fe8cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:59:00.051 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.040+0000 7f73fe8cf640 10 monclient(hunting): authenticate will time out at 1447.125805s 2024-03-29T11:59:00.052 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.045+0000 7f73f7fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f73f80a96c0 0x7f73f813f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54269/0 (socket says 172.21.15.98:54269) 2024-03-29T11:59:00.052 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.045+0000 7f73f7fff640 1 -- 172.21.15.98:0/1691185463 learned_addr learned my addr 172.21.15.98:0/1691185463 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:59:00.054 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.047+0000 7f73f6ffd640 1 -- 172.21.15.98:0/1691185463 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 716399126 0 0) 0x7f73f813fee0 con 0x7f73f80a96c0 2024-03-29T11:59:00.054 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.047+0000 7f73f6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:59:00.054 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.047+0000 7f73f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:59:00.055 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.047+0000 7f73f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:59:00.055 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.047+0000 7f73f6ffd640 10 monclient(hunting): my global_id is 4596 2024-03-29T11:59:00.055 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.047+0000 7f73f6ffd640 1 -- 172.21.15.98:0/1691185463 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73e0003520 con 0x7f73f80a96c0 2024-03-29T11:59:00.055 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.047+0000 7f73f6ffd640 1 -- 172.21.15.98:0/1691185463 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4031322187 0 0) 0x7f73f8145e80 con 0x7f73f80a9aa0 2024-03-29T11:59:00.055 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.047+0000 7f73f6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:59:00.055 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.047+0000 7f73f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:59:00.055 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.047+0000 7f73f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:59:00.055 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.047+0000 7f73f6ffd640 10 monclient(hunting): my global_id is 4643 2024-03-29T11:59:00.055 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.047+0000 7f73f6ffd640 1 -- 172.21.15.98:0/1691185463 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73f813fee0 con 0x7f73f80a9aa0 2024-03-29T11:59:00.058 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.048+0000 7f73f6ffd640 1 -- 172.21.15.98:0/1691185463 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 172892261 0 0) 0x7f73e0003520 con 0x7f73f80a96c0 2024-03-29T11:59:00.058 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.048+0000 7f73f6ffd640 1 -- 172.21.15.98:0/1691185463 >> v1:172.21.15.98:6790/0 conn(0x7f73f80a9aa0 legacy=0x7f73f8145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:59:00.058 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.048+0000 7f73f6ffd640 1 -- 172.21.15.98:0/1691185463 >> v1:172.21.15.121:6789/0 conn(0x7f73f80ad730 legacy=0x7f73f814dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:59:00.058 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.048+0000 7f73f6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:59:00.059 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.048+0000 7f73f6ffd640 1 monclient: found mon.noname-a 2024-03-29T11:59:00.059 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.048+0000 7f73f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:59:00.059 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.048+0000 7f73f6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T11:59:00.059 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.048+0000 7f73f6ffd640 1 -- 172.21.15.98:0/1691185463 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73f80b4440 con 0x7f73f80a96c0 2024-03-29T11:59:00.059 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.048+0000 7f73f6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:59:00.059 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.048+0000 7f73f6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:59:00.059 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.048+0000 7f73f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:59:00.059 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.048+0000 7f73fe8cf640 5 monclient: authenticate success, global_id 4596 2024-03-29T11:59:00.059 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.048+0000 7f73fe8cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:59:00.059 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.048+0000 7f73f6ffd640 1 -- 172.21.15.98:0/1691185463 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f73f0001a10 con 0x7f73f80a96c0 2024-03-29T11:59:00.059 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.048+0000 7f73f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:59:00.059 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.048+0000 7f73f6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:59:00.059 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.048+0000 7f73f6ffd640 10 monclient: dump: 2024-03-29T11:59:00.059 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:59:00.060 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:59:00.060 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:59:00.060 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:59:00.060 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:59:00.060 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:59:00.060 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:59:00.060 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:59:00.060 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:59:00.060 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:59:00.060 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.048+0000 7f73fe8cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:59:00.060 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.048+0000 7f73f6ffd640 1 -- 172.21.15.98:0/1691185463 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73f0001c30 con 0x7f73f80a96c0 2024-03-29T11:59:00.061 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.048+0000 7f73f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:59:00.061 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.048+0000 7f73fe8cf640 10 monclient: get_monmap_and_config success 2024-03-29T11:59:00.061 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.048+0000 7f73fe8cf640 4 set_mon_vals no callback set 2024-03-29T11:59:00.061 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.049+0000 7f73fe8cf640 10 monclient: shutdown 2024-03-29T11:59:00.061 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.049+0000 7f73fe8cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:59:00.061 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.049+0000 7f73fe8cf640 1 -- 172.21.15.98:0/1691185463 >> v1:172.21.15.98:6789/0 conn(0x7f73f80a96c0 legacy=0x7f73f813f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:59:00.061 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.050+0000 7f73fe8cf640 1 -- 172.21.15.98:0/1691185463 shutdown_connections 2024-03-29T11:59:00.061 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.050+0000 7f73fe8cf640 1 -- 172.21.15.98:0/1691185463 >> 172.21.15.98:0/1691185463 conn(0x7f73f80b35e0 msgr2=0x7f73f80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:59:00.061 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.050+0000 7f73fe8cf640 1 -- 172.21.15.98:0/1691185463 shutdown_connections 2024-03-29T11:59:00.061 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.050+0000 7f73fe8cf640 1 -- 172.21.15.98:0/1691185463 wait complete. 2024-03-29T11:59:00.061 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.051+0000 7f73fe8cf640 10 monclient: build_initial_monmap 2024-03-29T11:59:00.061 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.051+0000 7f73fe8cf640 10 monclient: monmap: 2024-03-29T11:59:00.061 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:59:00.061 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:59:00.062 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:59:00.062 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:59:00.062 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:59:00.062 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:59:00.062 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:59:00.062 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:59:00.062 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:59:00.062 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:59:00.063 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.052+0000 7f73fe8cf640 1 librados: starting msgr at 2024-03-29T11:59:00.063 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.052+0000 7f73fe8cf640 1 librados: starting objecter 2024-03-29T11:59:00.063 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.052+0000 7f73fe8cf640 1 Processor -- start 2024-03-29T11:59:00.064 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.052+0000 7f73fe8cf640 1 -- start start 2024-03-29T11:59:00.064 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.052+0000 7f73fe8cf640 1 librados: setting wanted keys 2024-03-29T11:59:00.064 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.052+0000 7f73fe8cf640 1 librados: calling monclient init 2024-03-29T11:59:00.064 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.052+0000 7f73fe8cf640 10 monclient: init 2024-03-29T11:59:00.064 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.053+0000 7f73fe8cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:59:00.064 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.053+0000 7f73fe8cf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:59:00.064 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.053+0000 7f73fe8cf640 10 monclient(hunting): picked mon.noname-b con 0x7f73f80a96c0 addr v1:172.21.15.121:6789/0 2024-03-29T11:59:00.064 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.053+0000 7f73fe8cf640 10 monclient(hunting): picked mon.noname-c con 0x7f73f80ad730 addr v1:172.21.15.98:6790/0 2024-03-29T11:59:00.066 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.056+0000 7f73fe8cf640 10 monclient(hunting): picked mon.noname-a con 0x7f73f81e4ae0 addr v1:172.21.15.98:6789/0 2024-03-29T11:59:00.066 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.059+0000 7f73fe8cf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73f81e4650 con 0x7f73f81e4ae0 2024-03-29T11:59:00.066 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.059+0000 7f73fe8cf640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73f813e850 con 0x7f73f80a96c0 2024-03-29T11:59:00.066 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.059+0000 7f73f7fff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f73f80a96c0 0x7f73f813d0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:37885/0 (socket says 172.21.15.98:37885) 2024-03-29T11:59:00.066 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.059+0000 7f73f7fff640 1 -- 172.21.15.98:0/2606380971 learned_addr learned my addr 172.21.15.98:0/2606380971 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:59:00.066 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.059+0000 7f73fe8cf640 1 -- 172.21.15.98:0/2606380971 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73f81411c0 con 0x7f73f80ad730 2024-03-29T11:59:00.066 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.059+0000 7f73fe8cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:59:00.066 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.059+0000 7f73fe8cf640 10 monclient(hunting): authenticate will time out at 1447.144663s 2024-03-29T11:59:00.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.059+0000 7f73f4ff9640 10 client.?.objecter ms_handle_connect 0x7f73f80a96c0 2024-03-29T11:59:00.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.059+0000 7f73f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:59:00.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.059+0000 7f73f4ff9640 1 -- 172.21.15.98:0/2606380971 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 320981958 0 0) 0x7f73f813e850 con 0x7f73f80a96c0 2024-03-29T11:59:00.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.059+0000 7f73f4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:59:00.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.059+0000 7f73f4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:59:00.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.059+0000 7f73f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:59:00.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.059+0000 7f73f4ff9640 10 monclient(hunting): my global_id is 4591 2024-03-29T11:59:00.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.059+0000 7f73f4ff9640 1 -- 172.21.15.98:0/2606380971 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73e4004eb0 con 0x7f73f80a96c0 2024-03-29T11:59:00.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.060+0000 7f73f4ff9640 1 -- 172.21.15.98:0/2606380971 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2723105562 0 0) 0x7f73e4004eb0 con 0x7f73f80a96c0 2024-03-29T11:59:00.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.060+0000 7f73f4ff9640 1 -- 172.21.15.98:0/2606380971 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73f813e850 con 0x7f73f80a96c0 2024-03-29T11:59:00.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.060+0000 7f73f4ff9640 1 -- 172.21.15.98:0/2606380971 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f73f0004d40 con 0x7f73f80a96c0 2024-03-29T11:59:00.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.060+0000 7f73f4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:59:00.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.060+0000 7f73f4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:59:00.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.060+0000 7f73f4ff9640 10 monclient(hunting): dump: 2024-03-29T11:59:00.071 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:59:00.071 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:59:00.071 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:59:00.071 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:59:00.071 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:59:00.072 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:59:00.072 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:59:00.072 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:59:00.072 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:59:00.072 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:59:00.072 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.060+0000 7f73f4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:59:00.072 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.060+0000 7f73f4ff9640 1 -- 172.21.15.98:0/2606380971 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3253185998 0 0) 0x7f73f813e850 con 0x7f73f80a96c0 2024-03-29T11:59:00.072 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.060+0000 7f73f4ff9640 1 -- 172.21.15.98:0/2606380971 >> v1:172.21.15.98:6790/0 conn(0x7f73f80ad730 legacy=0x7f73f813c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:59:00.072 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.060+0000 7f73f4ff9640 1 -- 172.21.15.98:0/2606380971 >> v1:172.21.15.98:6789/0 conn(0x7f73f81e4ae0 legacy=0x7f73f81e4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:59:00.072 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.060+0000 7f73f4ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:59:00.072 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.060+0000 7f73f4ff9640 1 monclient: found mon.b 2024-03-29T11:59:00.072 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.060+0000 7f73f4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:59:00.072 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.060+0000 7f73f4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:59:00.072 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.060+0000 7f73f4ff9640 1 -- 172.21.15.98:0/2606380971 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73f81f1170 con 0x7f73f80a96c0 2024-03-29T11:59:00.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.060+0000 7f73f4ff9640 10 monclient: _finish_auth 0 2024-03-29T11:59:00.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.060+0000 7f73f4ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:59:00.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.060+0000 7f73f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:59:00.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.061+0000 7f73f4ff9640 1 -- 172.21.15.98:0/2606380971 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73f00051c0 con 0x7f73f80a96c0 2024-03-29T11:59:00.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.061+0000 7f73f4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:59:00.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.061+0000 7f73f4ff9640 1 -- 172.21.15.98:0/2606380971 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f73f0005610 con 0x7f73f80a96c0 2024-03-29T11:59:00.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.061+0000 7f73f67fc640 4 set_mon_vals no callback set 2024-03-29T11:59:00.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.061+0000 7f73fe8cf640 5 monclient: authenticate success, global_id 4591 2024-03-29T11:59:00.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.061+0000 7f73fe8cf640 10 monclient: _renew_subs 2024-03-29T11:59:00.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.061+0000 7f73fe8cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:59:00.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.061+0000 7f73fe8cf640 1 -- 172.21.15.98:0/2606380971 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73f81f0160 con 0x7f73f80a96c0 2024-03-29T11:59:00.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.061+0000 7f73fe8cf640 10 client.4591.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:59:00.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.061+0000 7f73fe8cf640 10 monclient: _renew_subs 2024-03-29T11:59:00.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.061+0000 7f73fe8cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:59:00.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.061+0000 7f73fe8cf640 1 -- 172.21.15.98:0/2606380971 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f73f81f0640 con 0x7f73f80a96c0 2024-03-29T11:59:00.074 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.061+0000 7f73fe8cf640 1 librados: init done 2024-03-29T11:59:00.074 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.062+0000 7f73f4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:59:00.074 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.062+0000 7f73f4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:59:00.074 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.062+0000 7f73f4ff9640 10 monclient: dump: 2024-03-29T11:59:00.074 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:59:00.074 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:59:00.074 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:59:00.074 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:59:00.074 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:59:00.074 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:59:00.074 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:59:00.074 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:59:00.074 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:59:00.075 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:59:00.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.062+0000 7f73fe8cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:59:00.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.063+0000 7f73fe8cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:59:00.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.063+0000 7f73fe8cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:59:00.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.063+0000 7f73fe8cf640 1 -- 172.21.15.98:0/2606380971 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73f813d810 con 0x7f73f80a96c0 2024-03-29T11:59:00.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.063+0000 7f73f4ff9640 1 -- 172.21.15.98:0/2606380971 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f73f0001e20 con 0x7f73f80a96c0 2024-03-29T11:59:00.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.068+0000 7f73f4ff9640 1 -- 172.21.15.98:0/2606380971 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3033626086 0 0) 0x7f73f00850a0 con 0x7f73f80a96c0 2024-03-29T11:59:00.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.068+0000 7f73f4ff9640 10 client.4591.objecter ms_dispatch 0x7f73f8000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:59:00.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.068+0000 7f73f4ff9640 3 client.4591.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:59:00.076 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.068+0000 7f73f4ff9640 3 client.4591.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:59:00.076 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.068+0000 7f73f77fe640 10 monclient: get_auth_request con 0x7f73e406dd10 auth_method 0 2024-03-29T11:59:00.076 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.068+0000 7f73f4ff9640 20 client.4591.objecter dump_active .. 0 homeless 2024-03-29T11:59:00.076 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.068+0000 7f73f4ff9640 10 client.4591.objecter ms_handle_connect 0x7f73e406dd10 2024-03-29T11:59:00.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.072+0000 7f73f4ff9640 1 -- 172.21.15.98:0/2606380971 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f73f0050980 con 0x7f73f80a96c0 2024-03-29T11:59:00.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.072+0000 7f73f4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:59:00.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.072+0000 7f73f4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:59:00.402 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.393+0000 7f73fe8cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:59:00.402 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.393+0000 7f73fe8cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:59:00.402 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.393+0000 7f73fe8cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:59:00.402 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.393+0000 7f73fe8cf640 1 -- 172.21.15.98:0/2606380971 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73f8075f10 con 0x7f73f80a96c0 2024-03-29T11:59:00.403 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.396+0000 7f73f4ff9640 1 -- 172.21.15.98:0/2606380971 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3141891755) 0x7f73f00528b0 con 0x7f73f80a96c0 2024-03-29T11:59:00.403 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.396+0000 7f73f4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:59:00.403 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.396+0000 7f73f4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:59:00.404 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:59:00.404 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":11,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:58:22.774801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:58:22.774801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:59:00.407 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.400+0000 7f73fe8cf640 10 librados: watch_flush enter 2024-03-29T11:59:00.407 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.400+0000 7f73fe8cf640 10 librados: watch_flush exit 2024-03-29T11:59:00.407 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.400+0000 7f73fe8cf640 20 client.4591.objecter shutdown clearing up homeless session... 2024-03-29T11:59:00.407 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.400+0000 7f73fe8cf640 10 client.4591.objecter successfully canceled tick 2024-03-29T11:59:00.407 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.400+0000 7f73fe8cf640 1 -- 172.21.15.98:0/2606380971 >> v1:172.21.15.98:6816/4065186974 conn(0x7f73e406dd10 legacy=0x7f73e40701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:59:00.407 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.400+0000 7f73fe8cf640 10 monclient: shutdown 2024-03-29T11:59:00.407 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.400+0000 7f73fe8cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:59:00.407 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.400+0000 7f73fe8cf640 1 -- 172.21.15.98:0/2606380971 >> v1:172.21.15.121:6789/0 conn(0x7f73f80a96c0 legacy=0x7f73f813d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:59:00.409 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.402+0000 7f73fe8cf640 1 -- 172.21.15.98:0/2606380971 shutdown_connections 2024-03-29T11:59:00.409 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.402+0000 7f73fe8cf640 1 -- 172.21.15.98:0/2606380971 >> 172.21.15.98:0/2606380971 conn(0x7f73f80b35e0 msgr2=0x7f73f80b00b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:59:00.409 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.402+0000 7f73fe8cf640 1 -- 172.21.15.98:0/2606380971 shutdown_connections 2024-03-29T11:59:00.409 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.403+0000 7f73fe8cf640 1 -- 172.21.15.98:0/2606380971 wait complete. 2024-03-29T11:59:00.409 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:00.403+0000 7f73fe8cf640 1 librados: shutdown 2024-03-29T11:59:00.430 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:59:07.238 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:59:07.384 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.377+0000 7f47ca3a3640 10 monclient: get_monmap_and_config 2024-03-29T11:59:07.384 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.377+0000 7f47ca3a3640 10 monclient: build_initial_monmap 2024-03-29T11:59:07.384 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.377+0000 7f47ca3a3640 10 monclient: monmap: 2024-03-29T11:59:07.384 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:59:07.384 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:59:07.384 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:59:07.378337+0000 2024-03-29T11:59:07.384 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:59:07.378337+0000 2024-03-29T11:59:07.384 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:59:07.384 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:59:07.385 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:59:07.385 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:59:07.385 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:59:07.385 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:59:07.388 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.380+0000 7f47ca3a3640 1 Processor -- start 2024-03-29T11:59:07.388 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.380+0000 7f47ca3a3640 1 -- start start 2024-03-29T11:59:07.388 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.380+0000 7f47ca3a3640 10 monclient: init 2024-03-29T11:59:07.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.381+0000 7f47ca3a3640 10 monclient: _reopen_session rank -1 2024-03-29T11:59:07.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.381+0000 7f47ca3a3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:59:07.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.381+0000 7f47ca3a3640 10 monclient(hunting): picked mon.noname-a con 0x7f47c40b1f60 addr v1:172.21.15.98:6789/0 2024-03-29T11:59:07.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.381+0000 7f47ca3a3640 10 monclient(hunting): picked mon.noname-b con 0x7f47c4071eb0 addr v1:172.21.15.98:6790/0 2024-03-29T11:59:07.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.381+0000 7f47ca3a3640 10 monclient(hunting): picked mon.noname-c con 0x7f47c4143d10 addr v1:172.21.15.121:6789/0 2024-03-29T11:59:07.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.381+0000 7f47ca3a3640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47c406ef80 con 0x7f47c40b1f60 2024-03-29T11:59:07.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.381+0000 7f47ca3a3640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47c406f260 con 0x7f47c4143d10 2024-03-29T11:59:07.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.381+0000 7f47ca3a3640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47c40b40c0 con 0x7f47c4071eb0 2024-03-29T11:59:07.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.381+0000 7f47ca3a3640 10 monclient(hunting): _renew_subs 2024-03-29T11:59:07.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.381+0000 7f47ca3a3640 10 monclient(hunting): authenticate will time out at 1454.466348s 2024-03-29T11:59:07.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.381+0000 7f47c3fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f47c40b1f60 0x7f47c4143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:42350/0 (socket says 172.21.15.98:42350) 2024-03-29T11:59:07.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.381+0000 7f47c3fff640 1 -- 172.21.15.98:0/942935998 learned_addr learned my addr 172.21.15.98:0/942935998 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:59:07.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.382+0000 7f47c2ffd640 1 -- 172.21.15.98:0/942935998 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3209447373 0 0) 0x7f47c406ef80 con 0x7f47c40b1f60 2024-03-29T11:59:07.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.382+0000 7f47c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:59:07.390 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.382+0000 7f47c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:59:07.390 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.382+0000 7f47c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:59:07.390 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.382+0000 7f47c2ffd640 10 monclient(hunting): my global_id is 4599 2024-03-29T11:59:07.390 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.382+0000 7f47c2ffd640 1 -- 172.21.15.98:0/942935998 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47b0003520 con 0x7f47c40b1f60 2024-03-29T11:59:07.390 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.382+0000 7f47c2ffd640 1 -- 172.21.15.98:0/942935998 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1752859624 0 0) 0x7f47b0003520 con 0x7f47c40b1f60 2024-03-29T11:59:07.390 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.382+0000 7f47c2ffd640 1 -- 172.21.15.98:0/942935998 >> v1:172.21.15.98:6790/0 conn(0x7f47c4071eb0 legacy=0x7f47c4146470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:59:07.391 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.384+0000 7f47c2ffd640 1 -- 172.21.15.98:0/942935998 >> v1:172.21.15.121:6789/0 conn(0x7f47c4143d10 legacy=0x7f47c4146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:59:07.391 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.384+0000 7f47c2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:59:07.392 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.384+0000 7f47c2ffd640 1 monclient: found mon.noname-a 2024-03-29T11:59:07.392 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.384+0000 7f47c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:59:07.392 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.385+0000 7f47c2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T11:59:07.392 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.385+0000 7f47c2ffd640 1 -- 172.21.15.98:0/942935998 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47c4155d70 con 0x7f47c40b1f60 2024-03-29T11:59:07.392 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.385+0000 7f47c2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:59:07.392 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.385+0000 7f47c2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:59:07.392 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.385+0000 7f47c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:59:07.392 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.385+0000 7f47c2ffd640 1 -- 172.21.15.98:0/942935998 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f47bc001a10 con 0x7f47c40b1f60 2024-03-29T11:59:07.392 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.385+0000 7f47c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:59:07.392 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.385+0000 7f47c2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:59:07.393 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.385+0000 7f47c2ffd640 10 monclient: dump: 2024-03-29T11:59:07.393 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:59:07.393 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:59:07.393 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:59:07.393 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:59:07.393 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:59:07.393 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:59:07.393 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:59:07.393 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:59:07.393 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:59:07.393 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:59:07.393 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.385+0000 7f47ca3a3640 5 monclient: authenticate success, global_id 4599 2024-03-29T11:59:07.393 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.385+0000 7f47ca3a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:59:07.397 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.388+0000 7f47c2ffd640 1 -- 172.21.15.98:0/942935998 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47bc001c70 con 0x7f47c40b1f60 2024-03-29T11:59:07.397 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.388+0000 7f47c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:59:07.397 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.388+0000 7f47c2ffd640 1 -- 172.21.15.98:0/942935998 <== mon.0 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f47bc0020c0 con 0x7f47c40b1f60 2024-03-29T11:59:07.397 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.388+0000 7f47c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:59:07.397 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.388+0000 7f47c2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:59:07.397 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.388+0000 7f47c2ffd640 10 monclient: dump: 2024-03-29T11:59:07.397 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:59:07.397 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:59:07.397 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:59:07.397 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:59:07.397 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:59:07.397 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:59:07.398 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:59:07.398 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:59:07.398 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:59:07.398 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:59:07.398 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.388+0000 7f47ca3a3640 10 monclient: get_monmap_and_config success 2024-03-29T11:59:07.398 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.388+0000 7f47ca3a3640 4 set_mon_vals no callback set 2024-03-29T11:59:07.398 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.389+0000 7f47ca3a3640 10 monclient: shutdown 2024-03-29T11:59:07.398 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.389+0000 7f47ca3a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:59:07.398 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.389+0000 7f47ca3a3640 1 -- 172.21.15.98:0/942935998 >> v1:172.21.15.98:6789/0 conn(0x7f47c40b1f60 legacy=0x7f47c4143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:59:07.398 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.391+0000 7f47ca3a3640 1 -- 172.21.15.98:0/942935998 shutdown_connections 2024-03-29T11:59:07.399 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.391+0000 7f47ca3a3640 1 -- 172.21.15.98:0/942935998 >> 172.21.15.98:0/942935998 conn(0x7f47c4066a00 msgr2=0x7f47c4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:59:07.400 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.393+0000 7f47ca3a3640 1 -- 172.21.15.98:0/942935998 shutdown_connections 2024-03-29T11:59:07.401 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.394+0000 7f47ca3a3640 1 -- 172.21.15.98:0/942935998 wait complete. 2024-03-29T11:59:07.401 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.395+0000 7f47ca3a3640 10 monclient: build_initial_monmap 2024-03-29T11:59:07.401 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.395+0000 7f47ca3a3640 10 monclient: monmap: 2024-03-29T11:59:07.401 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:59:07.402 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:59:07.402 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:59:07.402 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:59:07.402 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:59:07.402 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:59:07.402 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:59:07.402 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:59:07.402 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:59:07.402 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:59:07.403 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.395+0000 7f47ca3a3640 1 librados: starting msgr at 2024-03-29T11:59:07.403 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.395+0000 7f47ca3a3640 1 librados: starting objecter 2024-03-29T11:59:07.403 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.395+0000 7f47ca3a3640 1 Processor -- start 2024-03-29T11:59:07.403 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.396+0000 7f47ca3a3640 1 -- start start 2024-03-29T11:59:07.404 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.396+0000 7f47ca3a3640 1 librados: setting wanted keys 2024-03-29T11:59:07.404 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.396+0000 7f47ca3a3640 1 librados: calling monclient init 2024-03-29T11:59:07.404 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.396+0000 7f47ca3a3640 10 monclient: init 2024-03-29T11:59:07.404 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.397+0000 7f47ca3a3640 10 monclient: _reopen_session rank -1 2024-03-29T11:59:07.404 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.397+0000 7f47ca3a3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:59:07.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.398+0000 7f47ca3a3640 10 monclient(hunting): picked mon.noname-b con 0x7f47c40b1f60 addr v1:172.21.15.121:6789/0 2024-03-29T11:59:07.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.399+0000 7f47c3fff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f47c40b1f60 0x7f47c4139310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.98:53887/0 (socket says 172.21.15.98:53887) 2024-03-29T11:59:07.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.399+0000 7f47c3fff640 1 -- 172.21.15.98:0/2242946336 learned_addr learned my addr 172.21.15.98:0/2242946336 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:59:07.407 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.400+0000 7f47ca3a3640 10 monclient(hunting): picked mon.noname-c con 0x7f47c4143d10 addr v1:172.21.15.98:6790/0 2024-03-29T11:59:07.407 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.400+0000 7f47c0ff9640 10 client.?.objecter ms_handle_connect 0x7f47c40b1f60 2024-03-29T11:59:07.408 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.400+0000 7f47c0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:59:07.408 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.400+0000 7f47ca3a3640 10 monclient(hunting): picked mon.noname-a con 0x7f47c41547a0 addr v1:172.21.15.98:6789/0 2024-03-29T11:59:07.408 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.400+0000 7f47c0ff9640 10 client.?.objecter ms_handle_connect 0x7f47c41547a0 2024-03-29T11:59:07.408 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.400+0000 7f47c0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:59:07.408 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.400+0000 7f47ca3a3640 1 -- 172.21.15.98:0/2242946336 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47c40ab8a0 con 0x7f47c41547a0 2024-03-29T11:59:07.408 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.401+0000 7f47ca3a3640 1 -- 172.21.15.98:0/2242946336 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47c40acca0 con 0x7f47c40b1f60 2024-03-29T11:59:07.408 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.401+0000 7f47ca3a3640 1 -- 172.21.15.98:0/2242946336 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47c4152840 con 0x7f47c4143d10 2024-03-29T11:59:07.408 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.401+0000 7f47ca3a3640 10 monclient(hunting): _renew_subs 2024-03-29T11:59:07.408 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.401+0000 7f47ca3a3640 10 monclient(hunting): authenticate will time out at 1454.486479s 2024-03-29T11:59:07.413 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.402+0000 7f47c0ff9640 1 -- 172.21.15.98:0/2242946336 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2952325539 0 0) 0x7f47c40ab8a0 con 0x7f47c41547a0 2024-03-29T11:59:07.413 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.402+0000 7f47c0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:59:07.413 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.402+0000 7f47c0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:59:07.413 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.402+0000 7f47c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:59:07.413 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.403+0000 7f47c0ff9640 10 monclient(hunting): my global_id is 4602 2024-03-29T11:59:07.413 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.403+0000 7f47c0ff9640 1 -- 172.21.15.98:0/2242946336 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47a4004eb0 con 0x7f47c41547a0 2024-03-29T11:59:07.413 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.403+0000 7f47c0ff9640 10 client.?.objecter ms_handle_connect 0x7f47c4143d10 2024-03-29T11:59:07.413 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.403+0000 7f47c0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:59:07.413 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.403+0000 7f47c0ff9640 1 -- 172.21.15.98:0/2242946336 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 128101887 0 0) 0x7f47c4152840 con 0x7f47c4143d10 2024-03-29T11:59:07.413 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.403+0000 7f47c0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:59:07.413 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.403+0000 7f47c0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:59:07.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.403+0000 7f47c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:59:07.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.403+0000 7f47c0ff9640 10 monclient(hunting): my global_id is 4646 2024-03-29T11:59:07.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.403+0000 7f47c0ff9640 1 -- 172.21.15.98:0/2242946336 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47c40ab8a0 con 0x7f47c4143d10 2024-03-29T11:59:07.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.403+0000 7f47c0ff9640 1 -- 172.21.15.98:0/2242946336 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3013036530 0 0) 0x7f47a4004eb0 con 0x7f47c41547a0 2024-03-29T11:59:07.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.403+0000 7f47c0ff9640 1 -- 172.21.15.98:0/2242946336 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47c4152840 con 0x7f47c41547a0 2024-03-29T11:59:07.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.403+0000 7f47c0ff9640 1 -- 172.21.15.98:0/2242946336 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f47ac001750 con 0x7f47c41547a0 2024-03-29T11:59:07.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.403+0000 7f47c0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:59:07.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.403+0000 7f47c0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:59:07.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.404+0000 7f47c0ff9640 10 monclient(hunting): dump: 2024-03-29T11:59:07.414 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:59:07.414 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:59:07.414 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:59:07.414 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:59:07.414 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:59:07.415 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:59:07.415 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:59:07.415 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:59:07.415 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:59:07.415 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:59:07.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.404+0000 7f47c0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:59:07.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.404+0000 7f47c0ff9640 1 -- 172.21.15.98:0/2242946336 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3391568650 0 0) 0x7f47c40ab8a0 con 0x7f47c4143d10 2024-03-29T11:59:07.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.404+0000 7f47c0ff9640 1 -- 172.21.15.98:0/2242946336 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47a4004eb0 con 0x7f47c4143d10 2024-03-29T11:59:07.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.404+0000 7f47c0ff9640 1 -- 172.21.15.98:0/2242946336 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1508893485 0 0) 0x7f47c4152840 con 0x7f47c41547a0 2024-03-29T11:59:07.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.404+0000 7f47c0ff9640 1 -- 172.21.15.98:0/2242946336 >> v1:172.21.15.98:6790/0 conn(0x7f47c4143d10 legacy=0x7f47c4139a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:59:07.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.404+0000 7f47c0ff9640 1 -- 172.21.15.98:0/2242946336 >> v1:172.21.15.121:6789/0 conn(0x7f47c40b1f60 legacy=0x7f47c4139310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:59:07.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.404+0000 7f47c0ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:59:07.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.404+0000 7f47c0ff9640 1 monclient: found mon.a 2024-03-29T11:59:07.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.404+0000 7f47c0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:59:07.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.404+0000 7f47c0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:59:07.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.404+0000 7f47c0ff9640 1 -- 172.21.15.98:0/2242946336 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47c41529d0 con 0x7f47c41547a0 2024-03-29T11:59:07.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.404+0000 7f47c0ff9640 10 monclient: _finish_auth 0 2024-03-29T11:59:07.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.404+0000 7f47c0ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:59:07.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.404+0000 7f47c0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:59:07.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.405+0000 7f47ca3a3640 5 monclient: authenticate success, global_id 4602 2024-03-29T11:59:07.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.405+0000 7f47ca3a3640 10 monclient: _renew_subs 2024-03-29T11:59:07.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.405+0000 7f47ca3a3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:59:07.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.405+0000 7f47ca3a3640 1 -- 172.21.15.98:0/2242946336 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47c40ab120 con 0x7f47c41547a0 2024-03-29T11:59:07.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.405+0000 7f47ca3a3640 10 client.4602.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:59:07.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.405+0000 7f47ca3a3640 10 monclient: _renew_subs 2024-03-29T11:59:07.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.405+0000 7f47ca3a3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:59:07.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.405+0000 7f47ca3a3640 1 -- 172.21.15.98:0/2242946336 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f47c4136520 con 0x7f47c41547a0 2024-03-29T11:59:07.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.405+0000 7f47ca3a3640 1 librados: init done 2024-03-29T11:59:07.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.407+0000 7f47c0ff9640 1 -- 172.21.15.98:0/2242946336 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47ac0018b0 con 0x7f47c41547a0 2024-03-29T11:59:07.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.407+0000 7f47c0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:59:07.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.407+0000 7f47c0ff9640 1 -- 172.21.15.98:0/2242946336 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7f47ac0035f0 con 0x7f47c41547a0 2024-03-29T11:59:07.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.407+0000 7f47c0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:59:07.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.407+0000 7f47c27fc640 4 set_mon_vals no callback set 2024-03-29T11:59:07.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.407+0000 7f47c0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:59:07.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.407+0000 7f47c0ff9640 10 monclient: dump: 2024-03-29T11:59:07.418 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:59:07.418 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:59:07.418 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:59:07.418 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:59:07.418 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:59:07.418 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:59:07.419 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:59:07.419 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:59:07.419 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:59:07.419 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:59:07.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.408+0000 7f47ba7fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:59:07.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.408+0000 7f47ba7fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:59:07.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.408+0000 7f47ba7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:59:07.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.408+0000 7f47ba7fc640 1 -- 172.21.15.98:0/2242946336 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47a8005080 con 0x7f47c41547a0 2024-03-29T11:59:07.420 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.413+0000 7f47c0ff9640 1 -- 172.21.15.98:0/2242946336 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7f47ac003750 con 0x7f47c41547a0 2024-03-29T11:59:07.421 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.414+0000 7f47c0ff9640 1 -- 172.21.15.98:0/2242946336 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3033626086 0 0) 0x7f47ac0835a0 con 0x7f47c41547a0 2024-03-29T11:59:07.421 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.414+0000 7f47c0ff9640 10 client.4602.objecter ms_dispatch 0x7f47c4000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:59:07.421 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.414+0000 7f47c0ff9640 3 client.4602.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:59:07.421 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.414+0000 7f47c0ff9640 3 client.4602.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:59:07.421 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.414+0000 7f47c0ff9640 20 client.4602.objecter dump_active .. 0 homeless 2024-03-29T11:59:07.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.415+0000 7f47c3fff640 10 monclient: get_auth_request con 0x7f47a406de80 auth_method 0 2024-03-29T11:59:07.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.415+0000 7f47c0ff9640 10 client.4602.objecter ms_handle_connect 0x7f47a406de80 2024-03-29T11:59:07.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.418+0000 7f47c0ff9640 1 -- 172.21.15.98:0/2242946336 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f47ac04ecb0 con 0x7f47c41547a0 2024-03-29T11:59:07.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.418+0000 7f47c0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:59:07.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.418+0000 7f47c0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:59:07.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.740+0000 7f47ba7fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:59:07.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.740+0000 7f47ba7fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:59:07.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.740+0000 7f47ba7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T11:59:07.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.740+0000 7f47ba7fc640 1 -- 172.21.15.98:0/2242946336 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47a8005330 con 0x7f47c41547a0 2024-03-29T11:59:07.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.740+0000 7f47c0ff9640 1 -- 172.21.15.98:0/2242946336 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3141891755) 0x7f47ac050db0 con 0x7f47c41547a0 2024-03-29T11:59:07.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.740+0000 7f47c0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:59:07.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.740+0000 7f47c0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:59:07.748 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:59:07.748 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":11,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:58:22.774801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:58:22.774801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:59:07.751 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.744+0000 7f47ba7fc640 10 librados: watch_flush enter 2024-03-29T11:59:07.751 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.744+0000 7f47ba7fc640 10 librados: watch_flush exit 2024-03-29T11:59:07.751 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.744+0000 7f47ba7fc640 20 client.4602.objecter shutdown clearing up homeless session... 2024-03-29T11:59:07.751 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.744+0000 7f47ba7fc640 10 client.4602.objecter successfully canceled tick 2024-03-29T11:59:07.751 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.744+0000 7f47ba7fc640 1 -- 172.21.15.98:0/2242946336 >> v1:172.21.15.98:6816/4065186974 conn(0x7f47a406de80 legacy=0x7f47a4070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:59:07.751 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.744+0000 7f47ba7fc640 10 monclient: shutdown 2024-03-29T11:59:07.751 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.744+0000 7f47ba7fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:59:07.751 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.744+0000 7f47ba7fc640 1 -- 172.21.15.98:0/2242946336 >> v1:172.21.15.98:6789/0 conn(0x7f47c41547a0 legacy=0x7f47c4135e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:59:07.752 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.744+0000 7f47ba7fc640 1 -- 172.21.15.98:0/2242946336 shutdown_connections 2024-03-29T11:59:07.752 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.744+0000 7f47ba7fc640 1 -- 172.21.15.98:0/2242946336 >> 172.21.15.98:0/2242946336 conn(0x7f47c4066a00 msgr2=0x7f47c40b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:59:07.752 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.745+0000 7f47ba7fc640 1 -- 172.21.15.98:0/2242946336 shutdown_connections 2024-03-29T11:59:07.752 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.745+0000 7f47ba7fc640 1 -- 172.21.15.98:0/2242946336 wait complete. 2024-03-29T11:59:07.752 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:07.745+0000 7f47ba7fc640 1 librados: shutdown 2024-03-29T11:59:07.775 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:59:14.583 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:59:14.734 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.726+0000 7fefdbfff640 10 monclient: get_monmap_and_config 2024-03-29T11:59:14.734 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.726+0000 7fefdbfff640 10 monclient: build_initial_monmap 2024-03-29T11:59:14.734 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.726+0000 7fefdbfff640 10 monclient: monmap: 2024-03-29T11:59:14.734 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:59:14.734 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:59:14.734 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:59:14.727376+0000 2024-03-29T11:59:14.734 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:59:14.727376+0000 2024-03-29T11:59:14.734 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:59:14.734 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:59:14.735 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:59:14.735 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:59:14.735 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T11:59:14.735 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:59:14.735 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.728+0000 7fefdbfff640 1 Processor -- start 2024-03-29T11:59:14.735 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.728+0000 7fefdbfff640 1 -- start start 2024-03-29T11:59:14.735 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.728+0000 7fefdbfff640 10 monclient: init 2024-03-29T11:59:14.737 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.729+0000 7fefdbfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:59:14.737 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.729+0000 7fefdbfff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:59:14.737 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.729+0000 7fefdbfff640 10 monclient(hunting): picked mon.noname-a con 0x7fefd4034850 addr v1:172.21.15.98:6789/0 2024-03-29T11:59:14.737 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.729+0000 7fefdbfff640 10 monclient(hunting): picked mon.noname-c con 0x7fefd40c9550 addr v1:172.21.15.121:6789/0 2024-03-29T11:59:14.737 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.729+0000 7fefdbfff640 10 monclient(hunting): picked mon.noname-b con 0x7fefd402aac0 addr v1:172.21.15.98:6790/0 2024-03-29T11:59:14.738 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.729+0000 7fefdbfff640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefd402c440 con 0x7fefd4034850 2024-03-29T11:59:14.738 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.729+0000 7fefdbfff640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefd402aed0 con 0x7fefd40c9550 2024-03-29T11:59:14.738 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.729+0000 7fefdbfff640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefd402b190 con 0x7fefd402aac0 2024-03-29T11:59:14.738 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.729+0000 7fefdbfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:59:14.738 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.729+0000 7fefdbfff640 10 monclient(hunting): authenticate will time out at 1461.814382s 2024-03-29T11:59:14.738 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.729+0000 7fefdb7fe640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fefd402aac0 0x7fefd40cc5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:56754/0 (socket says 172.21.15.98:56754) 2024-03-29T11:59:14.738 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.729+0000 7fefdb7fe640 1 -- 172.21.15.98:0/4217703596 learned_addr learned my addr 172.21.15.98:0/4217703596 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:59:14.738 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.729+0000 7fefd9ffb640 1 -- 172.21.15.98:0/4217703596 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4009098089 0 0) 0x7fefd402b190 con 0x7fefd402aac0 2024-03-29T11:59:14.738 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.729+0000 7fefd9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:59:14.738 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.729+0000 7fefd9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:59:14.738 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.729+0000 7fefd9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:59:14.738 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.729+0000 7fefd9ffb640 10 monclient(hunting): my global_id is 4649 2024-03-29T11:59:14.738 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.729+0000 7fefd9ffb640 1 -- 172.21.15.98:0/4217703596 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefc4003520 con 0x7fefd402aac0 2024-03-29T11:59:14.739 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.730+0000 7fefd9ffb640 1 -- 172.21.15.98:0/4217703596 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2848584861 0 0) 0x7fefc4003520 con 0x7fefd402aac0 2024-03-29T11:59:14.739 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.730+0000 7fefd9ffb640 1 -- 172.21.15.98:0/4217703596 >> v1:172.21.15.121:6789/0 conn(0x7fefd40c9550 legacy=0x7fefd40cbea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:59:14.739 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.730+0000 7fefd9ffb640 1 -- 172.21.15.98:0/4217703596 >> v1:172.21.15.98:6789/0 conn(0x7fefd4034850 legacy=0x7fefd40c8e40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:59:14.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.736+0000 7fefd9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:59:14.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.736+0000 7fefd9ffb640 1 monclient: found mon.noname-b 2024-03-29T11:59:14.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.736+0000 7fefd9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:59:14.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.736+0000 7fefd9ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T11:59:14.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.736+0000 7fefd9ffb640 1 -- 172.21.15.98:0/4217703596 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd4033060 con 0x7fefd402aac0 2024-03-29T11:59:14.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.736+0000 7fefd9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:59:14.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.736+0000 7fefd9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:59:14.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.736+0000 7fefd9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:59:14.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.736+0000 7fefd9ffb640 1 -- 172.21.15.98:0/4217703596 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fefd0001690 con 0x7fefd402aac0 2024-03-29T11:59:14.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.736+0000 7fefd9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:59:14.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.736+0000 7fefd9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:59:14.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.736+0000 7fefd9ffb640 10 monclient: dump: 2024-03-29T11:59:14.746 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:59:14.746 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:59:14.746 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:59:14.746 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:59:14.746 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:59:14.746 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:59:14.746 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:59:14.746 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:59:14.746 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:59:14.746 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:59:14.746 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.736+0000 7fefdbfff640 5 monclient: authenticate success, global_id 4649 2024-03-29T11:59:14.746 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.736+0000 7fefdbfff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:59:14.746 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.737+0000 7fefd9ffb640 1 -- 172.21.15.98:0/4217703596 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fefd0001920 con 0x7fefd402aac0 2024-03-29T11:59:14.746 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.737+0000 7fefd9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:59:14.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.737+0000 7fefd9ffb640 1 -- 172.21.15.98:0/4217703596 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fefd0001cb0 con 0x7fefd402aac0 2024-03-29T11:59:14.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.737+0000 7fefd9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:59:14.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.737+0000 7fefd9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:59:14.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.737+0000 7fefd9ffb640 10 monclient: dump: 2024-03-29T11:59:14.747 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:59:14.747 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:59:14.747 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:59:14.747 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:59:14.747 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:59:14.747 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:59:14.747 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:59:14.747 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:59:14.747 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:59:14.747 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:59:14.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.737+0000 7fefdbfff640 10 monclient: get_monmap_and_config success 2024-03-29T11:59:14.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.737+0000 7fefdbfff640 4 set_mon_vals no callback set 2024-03-29T11:59:14.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.738+0000 7fefdbfff640 10 monclient: shutdown 2024-03-29T11:59:14.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.738+0000 7fefdbfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:59:14.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.738+0000 7fefdbfff640 1 -- 172.21.15.98:0/4217703596 >> v1:172.21.15.98:6790/0 conn(0x7fefd402aac0 legacy=0x7fefd40cc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:59:14.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.738+0000 7fefdbfff640 1 -- 172.21.15.98:0/4217703596 shutdown_connections 2024-03-29T11:59:14.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.738+0000 7fefdbfff640 1 -- 172.21.15.98:0/4217703596 >> 172.21.15.98:0/4217703596 conn(0x7fefd40058f0 msgr2=0x7fefd4005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:59:14.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.738+0000 7fefdbfff640 1 -- 172.21.15.98:0/4217703596 shutdown_connections 2024-03-29T11:59:14.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.740+0000 7fefdbfff640 1 -- 172.21.15.98:0/4217703596 wait complete. 2024-03-29T11:59:14.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.741+0000 7fefdbfff640 10 monclient: build_initial_monmap 2024-03-29T11:59:14.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.741+0000 7fefdbfff640 10 monclient: monmap: 2024-03-29T11:59:14.749 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T11:59:14.749 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:59:14.749 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T11:59:14.749 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T11:59:14.749 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T11:59:14.749 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:59:14.749 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T11:59:14.749 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:59:14.750 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T11:59:14.750 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:59:14.750 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.742+0000 7fefdbfff640 1 librados: starting msgr at 2024-03-29T11:59:14.750 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.742+0000 7fefdbfff640 1 librados: starting objecter 2024-03-29T11:59:14.750 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.742+0000 7fefdbfff640 1 Processor -- start 2024-03-29T11:59:14.751 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.742+0000 7fefdbfff640 1 -- start start 2024-03-29T11:59:14.751 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.742+0000 7fefdbfff640 1 librados: setting wanted keys 2024-03-29T11:59:14.751 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.742+0000 7fefdbfff640 1 librados: calling monclient init 2024-03-29T11:59:14.751 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.742+0000 7fefdbfff640 10 monclient: init 2024-03-29T11:59:14.751 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.743+0000 7fefdbfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:59:14.751 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.743+0000 7fefdbfff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:59:14.752 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.745+0000 7fefdbfff640 10 monclient(hunting): picked mon.noname-b con 0x7fefd402aac0 addr v1:172.21.15.121:6789/0 2024-03-29T11:59:14.752 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.745+0000 7fefdbfff640 10 monclient(hunting): picked mon.noname-c con 0x7fefd4034850 addr v1:172.21.15.98:6790/0 2024-03-29T11:59:14.752 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.745+0000 7fefdbfff640 10 monclient(hunting): picked mon.noname-a con 0x7fefd40c9550 addr v1:172.21.15.98:6789/0 2024-03-29T11:59:14.752 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.745+0000 7fefdbfff640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefd4160c00 con 0x7fefd40c9550 2024-03-29T11:59:14.752 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.745+0000 7fefdbfff640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefd4161a80 con 0x7fefd402aac0 2024-03-29T11:59:14.752 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.745+0000 7fefdbfff640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefd40c1920 con 0x7fefd4034850 2024-03-29T11:59:14.752 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.745+0000 7fefdbfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:59:14.752 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.745+0000 7fefdbfff640 10 monclient(hunting): authenticate will time out at 1461.830624s 2024-03-29T11:59:14.753 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.745+0000 7fefda7fc640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fefd4034850 0x7fefd415df90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:58154/0 (socket says 172.21.15.98:58154) 2024-03-29T11:59:14.753 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.745+0000 7fefda7fc640 1 -- 172.21.15.98:0/3342148062 learned_addr learned my addr 172.21.15.98:0/3342148062 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T11:59:14.753 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.745+0000 7fefc37fe640 10 client.?.objecter ms_handle_connect 0x7fefd4034850 2024-03-29T11:59:14.753 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.745+0000 7fefc37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:59:14.753 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.745+0000 7fefc37fe640 1 -- 172.21.15.98:0/3342148062 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 418260911 0 0) 0x7fefd40c1920 con 0x7fefd4034850 2024-03-29T11:59:14.753 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.745+0000 7fefc37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:59:14.753 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.745+0000 7fefc37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:59:14.753 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.745+0000 7fefc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:59:14.753 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.746+0000 7fefc37fe640 10 monclient(hunting): my global_id is 4652 2024-03-29T11:59:14.753 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.746+0000 7fefc37fe640 1 -- 172.21.15.98:0/3342148062 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefc8004eb0 con 0x7fefd4034850 2024-03-29T11:59:14.756 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.748+0000 7fefc37fe640 10 client.?.objecter ms_handle_connect 0x7fefd402aac0 2024-03-29T11:59:14.756 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.748+0000 7fefc37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:59:14.757 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.748+0000 7fefc37fe640 1 -- 172.21.15.98:0/3342148062 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1745133924 0 0) 0x7fefd4161a80 con 0x7fefd402aac0 2024-03-29T11:59:14.757 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.748+0000 7fefc37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:59:14.757 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.748+0000 7fefc37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:59:14.757 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.748+0000 7fefc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:59:14.757 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.748+0000 7fefc37fe640 10 monclient(hunting): my global_id is 4597 2024-03-29T11:59:14.757 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.748+0000 7fefc37fe640 1 -- 172.21.15.98:0/3342148062 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefd40c1920 con 0x7fefd402aac0 2024-03-29T11:59:14.757 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.749+0000 7fefc37fe640 1 -- 172.21.15.98:0/3342148062 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2953382992 0 0) 0x7fefc8004eb0 con 0x7fefd4034850 2024-03-29T11:59:14.757 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.749+0000 7fefc37fe640 1 -- 172.21.15.98:0/3342148062 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fefd4161a80 con 0x7fefd4034850 2024-03-29T11:59:14.757 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.749+0000 7fefc37fe640 1 -- 172.21.15.98:0/3342148062 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fefdc04fdd0 con 0x7fefd4034850 2024-03-29T11:59:14.758 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.749+0000 7fefc37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:59:14.758 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.749+0000 7fefc37fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:59:14.758 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.749+0000 7fefc37fe640 10 monclient(hunting): dump: 2024-03-29T11:59:14.758 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:59:14.758 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:59:14.758 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:59:14.758 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:59:14.758 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:59:14.758 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:59:14.758 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:59:14.758 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:59:14.758 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:59:14.759 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:59:14.759 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.749+0000 7fefc37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:59:14.759 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.749+0000 7fefc37fe640 10 client.?.objecter ms_handle_connect 0x7fefd40c9550 2024-03-29T11:59:14.759 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.749+0000 7fefc37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:59:14.759 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.749+0000 7fefc37fe640 1 -- 172.21.15.98:0/3342148062 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3106673864 0 0) 0x7fefd4160c00 con 0x7fefd40c9550 2024-03-29T11:59:14.759 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.749+0000 7fefc37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:59:14.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.749+0000 7fefc37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:59:14.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.749+0000 7fefc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:59:14.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.749+0000 7fefc37fe640 10 monclient(hunting): my global_id is 4605 2024-03-29T11:59:14.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.749+0000 7fefc37fe640 1 -- 172.21.15.98:0/3342148062 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefc8004eb0 con 0x7fefd40c9550 2024-03-29T11:59:14.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.750+0000 7fefc37fe640 1 -- 172.21.15.98:0/3342148062 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2181929170 0 0) 0x7fefd4161a80 con 0x7fefd4034850 2024-03-29T11:59:14.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.750+0000 7fefc37fe640 1 -- 172.21.15.98:0/3342148062 >> v1:172.21.15.121:6789/0 conn(0x7fefd402aac0 legacy=0x7fefd4161370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:59:14.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.750+0000 7fefc37fe640 1 -- 172.21.15.98:0/3342148062 >> v1:172.21.15.98:6789/0 conn(0x7fefd40c9550 legacy=0x7fefd40bff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:59:14.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.750+0000 7fefc37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:59:14.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.750+0000 7fefc37fe640 1 monclient: found mon.c 2024-03-29T11:59:14.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.750+0000 7fefc37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:59:14.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.750+0000 7fefc37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:59:14.761 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.750+0000 7fefc37fe640 1 -- 172.21.15.98:0/3342148062 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd4160e50 con 0x7fefd4034850 2024-03-29T11:59:14.761 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.750+0000 7fefc37fe640 10 monclient: _finish_auth 0 2024-03-29T11:59:14.761 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.750+0000 7fefc37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:59:14.761 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.750+0000 7fefc37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:59:14.761 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.750+0000 7fefdbfff640 5 monclient: authenticate success, global_id 4652 2024-03-29T11:59:14.761 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.750+0000 7fefdbfff640 10 monclient: _renew_subs 2024-03-29T11:59:14.761 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.750+0000 7fefdbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:59:14.761 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.750+0000 7fefdbfff640 1 -- 172.21.15.98:0/3342148062 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefd415e6a0 con 0x7fefd4034850 2024-03-29T11:59:14.761 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.750+0000 7fefdbfff640 10 client.4652.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:59:14.761 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.750+0000 7fefdbfff640 10 monclient: _renew_subs 2024-03-29T11:59:14.761 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.750+0000 7fefdbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:59:14.762 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.750+0000 7fefdbfff640 1 -- 172.21.15.98:0/3342148062 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fefd415d9c0 con 0x7fefd4034850 2024-03-29T11:59:14.762 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.750+0000 7fefdbfff640 1 librados: init done 2024-03-29T11:59:14.762 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.751+0000 7fefdbfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:59:14.762 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.751+0000 7fefdbfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:59:14.762 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.751+0000 7fefdbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:59:14.762 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.751+0000 7fefdbfff640 1 -- 172.21.15.98:0/3342148062 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefd40be100 con 0x7fefd4034850 2024-03-29T11:59:14.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.752+0000 7fefc37fe640 1 -- 172.21.15.98:0/3342148062 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fefdc04f400 con 0x7fefd4034850 2024-03-29T11:59:14.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.752+0000 7fefc37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:59:14.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.752+0000 7fefd97fa640 4 set_mon_vals no callback set 2024-03-29T11:59:14.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.753+0000 7fefc37fe640 1 -- 172.21.15.98:0/3342148062 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 998563709 0 0) 0x7fefdc04f730 con 0x7fefd4034850 2024-03-29T11:59:14.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.753+0000 7fefc37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:59:14.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.753+0000 7fefc37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:59:14.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.753+0000 7fefc37fe640 10 monclient: dump: 2024-03-29T11:59:14.763 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T11:59:14.763 INFO:teuthology.misc.health.smithi098.stderr:fsid 683631e6-f6c8-470c-ab2f-82b5e5343501 2024-03-29T11:59:14.763 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T11:47:39.621520+0000 2024-03-29T11:59:14.763 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T11:47:39.621520+0000 2024-03-29T11:59:14.763 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T11:59:14.763 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T11:59:14.763 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T11:59:14.764 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:59:14.764 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T11:59:14.764 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T11:59:14.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.754+0000 7fefc37fe640 1 -- 172.21.15.98:0/3342148062 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1362299240 0 0) 0x7fefdc075710 con 0x7fefd4034850 2024-03-29T11:59:14.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.758+0000 7fefc37fe640 1 -- 172.21.15.98:0/3342148062 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3033626086 0 0) 0x7fefdc0ff300 con 0x7fefd4034850 2024-03-29T11:59:14.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.758+0000 7fefc37fe640 10 client.4652.objecter ms_dispatch 0x7fefdc000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:59:14.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.758+0000 7fefc37fe640 3 client.4652.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:59:14.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.758+0000 7fefc37fe640 3 client.4652.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:59:14.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.758+0000 7fefc37fe640 20 client.4652.objecter dump_active .. 0 homeless 2024-03-29T11:59:14.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.758+0000 7fefdaffd640 10 monclient: get_auth_request con 0x7fefc806d9c0 auth_method 0 2024-03-29T11:59:14.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.758+0000 7fefc37fe640 10 client.4652.objecter ms_handle_connect 0x7fefc806d9c0 2024-03-29T11:59:14.768 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.762+0000 7fefc37fe640 1 -- 172.21.15.98:0/3342148062 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fefdc0cabe0 con 0x7fefd4034850 2024-03-29T11:59:14.768 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.762+0000 7fefc37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:59:14.769 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:14.762+0000 7fefc37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:59:15.099 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:15.092+0000 7fefdbfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:59:15.099 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:15.092+0000 7fefdbfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:59:15.099 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:15.092+0000 7fefdbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T11:59:15.099 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:15.092+0000 7fefdbfff640 1 -- 172.21.15.98:0/3342148062 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fefd4160fe0 con 0x7fefd4034850 2024-03-29T11:59:15.100 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:15.093+0000 7fefc37fe640 1 -- 172.21.15.98:0/3342148062 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3141891755) 0x7fefdc0ccb10 con 0x7fefd4034850 2024-03-29T11:59:15.100 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:15.093+0000 7fefc37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:59:15.100 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:15.093+0000 7fefc37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:59:15.104 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T11:59:15.104 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":11,"fsid":"683631e6-f6c8-470c-ab2f-82b5e5343501","created":"2024-03-29T11:48:17.842517+0000","modified":"2024-03-29T11:58:22.774801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:58:22.774801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3764135-6315-4c21-b430-e3515422b4a5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"c3a0e291-f5e3-4542-b78b-003929905548","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"c3d2484d-a574-4ad6-8210-6b4ff2adde91","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c6c10299-76b6-4de5-ac71-6de9c36b6de4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0156b67c-8892-45ea-8bff-c2e991ee33a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"510d2deb-540c-43f6-8d32-6783917c2208","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bcd19919-e170-47a2-8b81-bcd5931571fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b0600d-02cc-4677-9176-be8248744f5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:59:15.107 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:15.100+0000 7fefdbfff640 10 librados: watch_flush enter 2024-03-29T11:59:15.107 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:15.100+0000 7fefdbfff640 10 librados: watch_flush exit 2024-03-29T11:59:15.107 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:15.100+0000 7fefdbfff640 20 client.4652.objecter shutdown clearing up homeless session... 2024-03-29T11:59:15.107 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:15.100+0000 7fefdbfff640 10 client.4652.objecter successfully canceled tick 2024-03-29T11:59:15.108 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:15.101+0000 7fefdbfff640 1 -- 172.21.15.98:0/3342148062 >> v1:172.21.15.98:6816/4065186974 conn(0x7fefc806d9c0 legacy=0x7fefc806fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:59:15.108 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:15.101+0000 7fefdbfff640 10 monclient: shutdown 2024-03-29T11:59:15.108 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:15.101+0000 7fefdbfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:59:15.108 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:15.101+0000 7fefdbfff640 1 -- 172.21.15.98:0/3342148062 >> v1:172.21.15.98:6790/0 conn(0x7fefd4034850 legacy=0x7fefd415df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:59:15.109 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:15.102+0000 7fefdbfff640 1 -- 172.21.15.98:0/3342148062 shutdown_connections 2024-03-29T11:59:15.109 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:15.102+0000 7fefdbfff640 1 -- 172.21.15.98:0/3342148062 >> 172.21.15.98:0/3342148062 conn(0x7fefd40058f0 msgr2=0x7fefd40043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:59:15.109 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:15.102+0000 7fefdbfff640 1 -- 172.21.15.98:0/3342148062 shutdown_connections 2024-03-29T11:59:15.109 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:15.102+0000 7fefdbfff640 1 -- 172.21.15.98:0/3342148062 wait complete. 2024-03-29T11:59:15.109 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T11:59:15.102+0000 7fefdbfff640 1 librados: shutdown 2024-03-29T11:59:15.126 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:59:15.126 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:59:15.131 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T11:59:15.131 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T11:59:15.131 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:59:15.168 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T11:59:15.168 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T11:59:15.168 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:59:15.243 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T11:59:15.243 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T11:59:15.243 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:59:15.314 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T11:59:15.314 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T11:59:15.314 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:59:15.382 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T11:59:15.382 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T11:59:15.382 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:59:15.417 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T11:59:15.417 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T11:59:15.418 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:59:15.487 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T11:59:15.488 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T11:59:15.488 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:59:15.555 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T11:59:15.555 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T11:59:15.556 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:59:15.631 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T11:59:15.631 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T11:59:15.631 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T11:59:15.632 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:59:15.656 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T11:59:15.656 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T11:59:15.656 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:59:15.695 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T11:59:15.696 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T11:59:15.696 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T11:59:15.696 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:59:15.707 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T11:59:15.708 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T11:59:15.708 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:59:15.759 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T11:59:15.759 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T11:59:15.759 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:59:15.770 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T11:59:15.770 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T11:59:15.770 DEBUG:teuthology.orchestra.run.smithi098:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T11:59:15.833 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi098.front.sepia.ceph.com 2024-03-29T11:59:15.833 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T11:59:18.731 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi098.front.sepia.ceph.com 2024-03-29T11:59:18.731 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T11:59:18.871 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi098.front.sepia.ceph.com 2024-03-29T11:59:18.872 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T11:59:18.943 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi098.front.sepia.ceph.com 2024-03-29T11:59:18.943 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T11:59:19.000 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi121.front.sepia.ceph.com 2024-03-29T11:59:19.000 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T11:59:19.325 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi121.front.sepia.ceph.com 2024-03-29T11:59:19.325 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T11:59:19.419 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi121.front.sepia.ceph.com 2024-03-29T11:59:19.419 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T11:59:19.503 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi121.front.sepia.ceph.com 2024-03-29T11:59:19.503 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T11:59:19.560 INFO:tasks.ceph:Archiving mon data... 2024-03-29T11:59:19.562 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622735/data/mon.a.tgz 2024-03-29T11:59:19.562 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-03-29T11:59:19.577 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.BYTglbafry 2024-03-29T11:59:19.577 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.BYTglbafry 2024-03-29T11:59:19.693 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.BYTglbafry 2024-03-29T11:59:19.778 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.BYTglbafry is 211KB 2024-03-29T11:59:19.872 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.BYTglbafry 2024-03-29T11:59:19.886 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622735/data/mon.c.tgz 2024-03-29T11:59:19.886 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-03-29T11:59:19.939 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.ecdvan2V7a 2024-03-29T11:59:19.940 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.ecdvan2V7a 2024-03-29T11:59:20.041 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.ecdvan2V7a 2024-03-29T11:59:20.127 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.ecdvan2V7a is 211KB 2024-03-29T11:59:20.221 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.ecdvan2V7a 2024-03-29T11:59:20.236 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622735/data/mon.b.tgz 2024-03-29T11:59:20.236 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2024-03-29T11:59:20.251 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.KBESzzJRHr 2024-03-29T11:59:20.251 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.KBESzzJRHr 2024-03-29T11:59:20.369 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.KBESzzJRHr 2024-03-29T11:59:20.455 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.KBESzzJRHr is 211KB 2024-03-29T11:59:20.548 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.KBESzzJRHr 2024-03-29T11:59:20.562 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T11:59:20.563 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:59:20.565 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:59:20.647 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:59:20.683 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:59:20.683 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T11:59:20.685 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622735/remote/smithi098/crash 2024-03-29T11:59:20.686 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:59:20.714 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622735/remote/smithi121/crash 2024-03-29T11:59:20.715 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:59:20.749 INFO:tasks.ceph:Compressing logs... 2024-03-29T11:59:20.749 DEBUG:teuthology.orchestra.run.smithi098:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:59:20.760 DEBUG:teuthology.orchestra.run.smithi121:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:59:20.823 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38817.log 2024-03-29T11:59:20.823 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41082.log 2024-03-29T11:59:20.824 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38583.log 2024-03-29T11:59:20.824 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40104.log 2024-03-29T11:59:20.824 INFO:teuthology.orchestra.run.smithi098.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38817.log.gz 2024-03-29T11:59:20.824 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38583.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41082.log.gz 2024-03-29T11:59:20.824 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T11:59:20.824 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38463.log 2024-03-29T11:59:20.824 INFO:teuthology.orchestra.run.smithi098.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.38583.log.gz 2024-03-29T11:59:20.824 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40026.log 2024-03-29T11:59:20.824 INFO:teuthology.orchestra.run.smithi098.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40104.log.gz 2024-03-29T11:59:20.824 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40224.log 2024-03-29T11:59:20.824 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38032.log 2024-03-29T11:59:20.824 INFO:teuthology.orchestra.run.smithi098.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38463.log.gz 2024-03-29T11:59:20.824 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41043.log 2024-03-29T11:59:20.825 INFO:teuthology.orchestra.run.smithi098.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40026.log.gz 2024-03-29T11:59:20.825 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40065.log 2024-03-29T11:59:20.825 INFO:teuthology.orchestra.run.smithi098.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40224.log.gz 2024-03-29T11:59:20.825 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39870.log 2024-03-29T11:59:20.825 INFO:teuthology.orchestra.run.smithi098.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38032.log.gz 2024-03-29T11:59:20.825 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40065.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41043.log.gz 2024-03-29T11:59:20.825 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40263.log 2024-03-29T11:59:20.825 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34307.log 2024-03-29T11:59:20.825 INFO:teuthology.orchestra.run.smithi098.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40065.log.gz 2024-03-29T11:59:20.825 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40692.log 2024-03-29T11:59:20.825 INFO:teuthology.orchestra.run.smithi098.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39870.log.gz 2024-03-29T11:59:20.825 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34307.log: gzip -5 --verbose -- 85.8% /var/log/ceph/ceph-client.admin.39051.log 2024-03-29T11:59:20.825 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40263.log.gz 2024-03-29T11:59:20.825 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40887.log 2024-03-29T11:59:20.826 INFO:teuthology.orchestra.run.smithi098.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.34307.log.gz 2024-03-29T11:59:20.826 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41238.log 2024-03-29T11:59:20.826 INFO:teuthology.orchestra.run.smithi098.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40692.log.gz 2024-03-29T11:59:20.826 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40887.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39051.log.gz 2024-03-29T11:59:20.826 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40536.log 2024-03-29T11:59:20.826 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38778.log 2024-03-29T11:59:20.826 INFO:teuthology.orchestra.run.smithi098.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40887.log.gz 2024-03-29T11:59:20.826 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38739.log 2024-03-29T11:59:20.826 INFO:teuthology.orchestra.run.smithi098.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41238.log.gz 2024-03-29T11:59:20.826 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38778.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40536.log.gz 2024-03-29T11:59:20.826 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40145.log 2024-03-29T11:59:20.826 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40731.log 2024-03-29T11:59:20.826 INFO:teuthology.orchestra.run.smithi098.stderr: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.38778.log.gz 2024-03-29T11:59:20.826 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40926.log 2024-03-29T11:59:20.827 INFO:teuthology.orchestra.run.smithi098.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38739.log.gz 2024-03-29T11:59:20.827 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40731.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40145.log.gz 2024-03-29T11:59:20.827 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38071.log 2024-03-29T11:59:20.827 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38268.log 2024-03-29T11:59:20.827 INFO:teuthology.orchestra.run.smithi098.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40731.log.gz 2024-03-29T11:59:20.827 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38188.log 2024-03-29T11:59:20.827 INFO:teuthology.orchestra.run.smithi098.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40926.log.gz 2024-03-29T11:59:20.827 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39675.log 2024-03-29T11:59:20.827 INFO:teuthology.orchestra.run.smithi098.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38071.log.gz 2024-03-29T11:59:20.827 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38188.log: gzip -5 --verbose -- 85.7% /var/log/ceph/ceph-client.admin.41277.log -- replaced with /var/log/ceph/ceph-client.admin.38268.log.gz 2024-03-29T11:59:20.827 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-29T11:59:20.827 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38149.log 2024-03-29T11:59:20.827 INFO:teuthology.orchestra.run.smithi098.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38188.log.gz 2024-03-29T11:59:20.827 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41277.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39675.log.gz 2024-03-29T11:59:20.828 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37993.log 2024-03-29T11:59:20.828 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38661.log 2024-03-29T11:59:20.828 INFO:teuthology.orchestra.run.smithi098.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41277.log.gz 2024-03-29T11:59:20.828 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39246.log 2024-03-29T11:59:20.828 INFO:teuthology.orchestra.run.smithi098.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38149.log.gz 2024-03-29T11:59:20.828 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38661.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:59:20.828 INFO:teuthology.orchestra.run.smithi098.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.37993.log.gz 2024-03-29T11:59:20.828 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39636.log 2024-03-29T11:59:20.828 INFO:teuthology.orchestra.run.smithi098.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38661.log.gz 2024-03-29T11:59:20.828 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph.audit.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39246.log.gz 2024-03-29T11:59:20.828 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39363.log 2024-03-29T11:59:20.828 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38973.log 2024-03-29T11:59:20.829 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39636.log: 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:59:20.829 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38622.log 2024-03-29T11:59:20.829 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41199.log 2024-03-29T11:59:20.829 INFO:teuthology.orchestra.run.smithi098.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39636.log.gz 2024-03-29T11:59:20.829 INFO:teuthology.orchestra.run.smithi098.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39363.log.gz 2024-03-29T11:59:20.829 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34593.log 2024-03-29T11:59:20.829 INFO:teuthology.orchestra.run.smithi098.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38973.log.gz 2024-03-29T11:59:20.829 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39792.log 2024-03-29T11:59:20.829 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41199.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38622.log.gz 2024-03-29T11:59:20.829 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40380.log 2024-03-29T11:59:20.829 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39792.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41199.log.gz 2024-03-29T11:59:20.829 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40770.log 2024-03-29T11:59:20.830 INFO:teuthology.orchestra.run.smithi098.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34593.log.gz 2024-03-29T11:59:20.830 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40380.log: 86.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.39792.log.gz 2024-03-29T11:59:20.830 INFO:teuthology.orchestra.run.smithi098.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40809.log 2024-03-29T11:59:20.830 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40770.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40380.log.gz 2024-03-29T11:59:20.830 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41121.log 2024-03-29T11:59:20.830 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40497.log 2024-03-29T11:59:20.830 INFO:teuthology.orchestra.run.smithi098.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40770.log.gz 2024-03-29T11:59:20.830 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39753.log 2024-03-29T11:59:20.830 INFO:teuthology.orchestra.run.smithi098.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40809.log.gz 2024-03-29T11:59:20.830 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39714.log 2024-03-29T11:59:20.830 INFO:teuthology.orchestra.run.smithi098.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41121.log.gz 2024-03-29T11:59:20.830 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39753.log: gzip -5 85.8% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.40497.log.gz 2024-03-29T11:59:20.831 INFO:teuthology.orchestra.run.smithi098.stderr: /var/log/ceph/ceph-osd.1.log 2024-03-29T11:59:20.831 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39714.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39753.log.gz 2024-03-29T11:59:20.831 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38856.log 2024-03-29T11:59:20.831 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38502.log 2024-03-29T11:59:20.831 INFO:teuthology.orchestra.run.smithi098.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39714.log.gz 2024-03-29T11:59:20.831 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40653.log 2024-03-29T11:59:20.831 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39480.log 2024-03-29T11:59:20.831 INFO:teuthology.orchestra.run.smithi098.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.38856.log.gz 2024-03-29T11:59:20.831 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40653.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38502.log.gz 2024-03-29T11:59:20.831 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40302.log 2024-03-29T11:59:20.831 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose 86.1% -- -- replaced with /var/log/ceph/ceph-client.admin.40653.log.gz 2024-03-29T11:59:20.831 INFO:teuthology.orchestra.run.smithi098.stderr: /var/log/ceph/ceph-client.admin.38385.log 2024-03-29T11:59:20.832 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39480.log: /var/log/ceph/ceph-client.admin.40302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34696.log 2024-03-29T11:59:20.832 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38385.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39480.log.gz 2024-03-29T11:59:20.832 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39987.log 2024-03-29T11:59:20.832 INFO:teuthology.orchestra.run.smithi098.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40302.log.gz 2024-03-29T11:59:20.832 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39519.log 2024-03-29T11:59:20.832 INFO:teuthology.orchestra.run.smithi098.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38385.log.gz 2024-03-29T11:59:20.832 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39402.log 2024-03-29T11:59:20.832 INFO:teuthology.orchestra.run.smithi098.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34696.log.gz 2024-03-29T11:59:20.832 INFO:teuthology.orchestra.run.smithi098.stderr:gzip/var/log/ceph/ceph-client.admin.39519.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39831.log 2024-03-29T11:59:20.832 INFO:teuthology.orchestra.run.smithi098.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39987.log.gz 2024-03-29T11:59:20.832 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38541.log 2024-03-29T11:59:20.832 INFO:teuthology.orchestra.run.smithi098.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39519.log.gz 2024-03-29T11:59:20.832 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38700.log 2024-03-29T11:59:20.832 INFO:teuthology.orchestra.run.smithi098.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39402.log.gz 2024-03-29T11:59:20.833 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38541.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39831.log.gz 2024-03-29T11:59:20.833 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41355.log 2024-03-29T11:59:20.833 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40185.log 2024-03-29T11:59:20.833 INFO:teuthology.orchestra.run.smithi098.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38541.log.gz 2024-03-29T11:59:20.833 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41355.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T11:59:20.833 INFO:teuthology.orchestra.run.smithi098.stderr: 86.9% 85.6% -- replaced with /var/log/ceph/ceph-client.admin.38700.log.gz 2024-03-29T11:59:20.833 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41355.log.gz 2024-03-29T11:59:20.833 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39909.log 2024-03-29T11:59:20.833 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40419.log 2024-03-29T11:59:20.833 INFO:teuthology.orchestra.run.smithi098.stderr: 85.7%/var/log/ceph/ceph-osd.3.log: -- replaced with /var/log/ceph/ceph-client.admin.40185.log.gz 2024-03-29T11:59:20.833 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39012.log 2024-03-29T11:59:20.833 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38424.log 2024-03-29T11:59:20.833 INFO:teuthology.orchestra.run.smithi098.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39909.log.gz 2024-03-29T11:59:20.833 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41160.log 2024-03-29T11:59:20.833 INFO:teuthology.orchestra.run.smithi098.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40419.log.gz 2024-03-29T11:59:20.833 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38424.log: 87.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.39012.log.gz -5 2024-03-29T11:59:20.834 INFO:teuthology.orchestra.run.smithi098.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39207.log 2024-03-29T11:59:20.834 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39441.log 2024-03-29T11:59:20.834 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41160.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38424.log.gz 2024-03-29T11:59:20.834 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38307.log 2024-03-29T11:59:20.834 INFO:teuthology.orchestra.run.smithi098.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41160.log.gz 2024-03-29T11:59:20.834 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39441.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T11:59:20.834 INFO:teuthology.orchestra.run.smithi098.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39207.log.gz 2024-03-29T11:59:20.834 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38307.log: gzip 86.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.39441.log.gz --verbose 2024-03-29T11:59:20.834 INFO:teuthology.orchestra.run.smithi098.stderr: -- /var/log/ceph/ceph-client.admin.38934.log 2024-03-29T11:59:20.834 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41004.log 2024-03-29T11:59:20.834 INFO:teuthology.orchestra.run.smithi098.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38307.log.gz 2024-03-29T11:59:20.834 INFO:teuthology.orchestra.run.smithi098.stderr:gzip/var/log/ceph/ceph-client.admin.38934.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41316.log 2024-03-29T11:59:20.834 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41004.log: gzip -5 --verbose 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38934.log.gz 2024-03-29T11:59:20.834 INFO:teuthology.orchestra.run.smithi098.stderr: -- /var/log/ceph/ceph-client.admin.40965.log 2024-03-29T11:59:20.834 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41316.log: gzip 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41004.log.gz 2024-03-29T11:59:20.835 INFO:teuthology.orchestra.run.smithi098.stderr: -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-29T11:59:20.835 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40965.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41316.log.gz 2024-03-29T11:59:20.835 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38895.log 2024-03-29T11:59:20.835 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-osd.0.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40965.log.gz 2024-03-29T11:59:20.835 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40458.log 2024-03-29T11:59:20.835 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40341.log 2024-03-29T11:59:20.835 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40458.log: gzip 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38895.log.gz -5 2024-03-29T11:59:20.835 INFO:teuthology.orchestra.run.smithi098.stderr: --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:59:20.835 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38346.log 2024-03-29T11:59:20.835 INFO:teuthology.orchestra.run.smithi098.stderr: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.40458.log.gz 2024-03-29T11:59:20.835 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40341.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38110.log 2024-03-29T11:59:20.835 INFO:teuthology.orchestra.run.smithi098.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40341.log.gz/var/log/ceph/ceph-client.admin.38346.log: 2024-03-29T11:59:20.835 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T11:59:20.835 INFO:teuthology.orchestra.run.smithi098.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38346.log.gz 2024-03-29T11:59:20.836 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38110.log: gzip -5 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:59:20.836 INFO:teuthology.orchestra.run.smithi098.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39129.log 2024-03-29T11:59:20.836 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39597.log 2024-03-29T11:59:20.836 INFO:teuthology.orchestra.run.smithi098.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38110.log.gz 2024-03-29T11:59:20.836 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40575.log 2024-03-29T11:59:20.836 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39129.log: /var/log/ceph/ceph-client.admin.39597.log: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.39129.log.gz 2024-03-29T11:59:20.836 INFO:teuthology.orchestra.run.smithi098.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39597.log.gz 2024-03-29T11:59:20.836 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39948.log 2024-03-29T11:59:20.836 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39168.log 2024-03-29T11:59:20.836 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39948.log: /var/log/ceph/ceph-client.admin.40575.log: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.40575.log.gz 2024-03-29T11:59:20.836 INFO:teuthology.orchestra.run.smithi098.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39948.log.gz 2024-03-29T11:59:20.836 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39558.log 2024-03-29T11:59:20.836 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40848.log 2024-03-29T11:59:20.836 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39558.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39168.log.gz 2024-03-29T11:59:20.836 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41394.log 2024-03-29T11:59:20.837 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40848.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39558.log.gz 2024-03-29T11:59:20.837 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39324.log 2024-03-29T11:59:20.837 INFO:teuthology.orchestra.run.smithi098.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40848.log.gz 2024-03-29T11:59:20.837 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38227.log 2024-03-29T11:59:20.837 INFO:teuthology.orchestra.run.smithi098.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41394.log.gz 2024-03-29T11:59:20.837 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40614.log 2024-03-29T11:59:20.837 INFO:teuthology.orchestra.run.smithi098.stderr: 86.3%/var/log/ceph/ceph-client.admin.38227.log: -- replaced with /var/log/ceph/ceph-client.admin.39324.log.gz 2024-03-29T11:59:20.837 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39090.log 2024-03-29T11:59:20.837 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40614.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38227.log.gz 2024-03-29T11:59:20.837 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41433.log 2024-03-29T11:59:20.837 INFO:teuthology.orchestra.run.smithi098.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40614.log.gz 2024-03-29T11:59:20.837 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39090.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28482.log 2024-03-29T11:59:20.837 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41433.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39090.log.gz 2024-03-29T11:59:20.837 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39285.log 2024-03-29T11:59:20.837 INFO:teuthology.orchestra.run.smithi098.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41433.log.gz 2024-03-29T11:59:20.838 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph.tmp-client.admin.28482.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T11:59:20.838 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39285.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28482.log.gz 2024-03-29T11:59:20.838 INFO:teuthology.orchestra.run.smithi098.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39285.log.gz 2024-03-29T11:59:20.843 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T11:59:20.843 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34899.log 2024-03-29T11:59:20.844 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35019.log 2024-03-29T11:59:20.844 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.34899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35059.log 2024-03-29T11:59:20.844 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35019.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:59:20.844 INFO:teuthology.orchestra.run.smithi121.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34899.log.gz 2024-03-29T11:59:20.844 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35059.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T11:59:20.844 INFO:teuthology.orchestra.run.smithi121.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35019.log.gz 2024-03-29T11:59:20.844 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34779.log 2024-03-29T11:59:20.844 INFO:teuthology.orchestra.run.smithi121.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35059.log.gz 2024-03-29T11:59:20.844 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34819.log 2024-03-29T11:59:20.844 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.34779.log: 88.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:59:20.844 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T11:59:20.844 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.34819.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34779.log.gzgzip 2024-03-29T11:59:20.844 INFO:teuthology.orchestra.run.smithi121.stderr: -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T11:59:20.845 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34939.log 2024-03-29T11:59:20.845 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-osd.7.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34819.log.gz 2024-03-29T11:59:20.845 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T11:59:20.845 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.34939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34859.log 2024-03-29T11:59:20.845 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:59:20.845 INFO:teuthology.orchestra.run.smithi121.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34939.log.gz 2024-03-29T11:59:20.845 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.34859.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34859.log.gz 2024-03-29T11:59:20.845 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34979.log 2024-03-29T11:59:20.845 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T11:59:20.845 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.34979.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34979.log.gz 2024-03-29T11:59:20.845 INFO:teuthology.orchestra.run.smithi121.stderr: 87.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:59:20.845 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-mon.b.log: 93.9% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T11:59:20.877 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-mon.c.log: 92.6% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T11:59:32.972 INFO:teuthology.orchestra.run.smithi121.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T11:59:33.083 INFO:teuthology.orchestra.run.smithi121.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T11:59:33.107 INFO:teuthology.orchestra.run.smithi121.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T11:59:35.591 INFO:teuthology.orchestra.run.smithi098.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T11:59:36.194 INFO:teuthology.orchestra.run.smithi121.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T11:59:37.939 INFO:teuthology.orchestra.run.smithi098.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T11:59:38.745 INFO:teuthology.orchestra.run.smithi098.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T11:59:38.836 INFO:teuthology.orchestra.run.smithi098.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T12:00:11.811 INFO:teuthology.orchestra.run.smithi098.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T12:00:12.505 INFO:teuthology.orchestra.run.smithi098.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T12:00:12.507 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-29T12:00:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:real 0m51.736s 2024-03-29T12:00:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:user 2m44.510s 2024-03-29T12:00:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:sys 0m5.776s 2024-03-29T12:00:30.684 INFO:teuthology.orchestra.run.smithi121.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T12:00:30.685 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T12:00:30.685 INFO:teuthology.orchestra.run.smithi121.stderr:real 1m9.876s 2024-03-29T12:00:30.686 INFO:teuthology.orchestra.run.smithi121.stderr:user 1m57.316s 2024-03-29T12:00:30.686 INFO:teuthology.orchestra.run.smithi121.stderr:sys 0m4.403s 2024-03-29T12:00:30.686 INFO:tasks.ceph:Archiving logs... 2024-03-29T12:00:30.686 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622735/remote/smithi098/log 2024-03-29T12:00:30.688 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T12:00:54.173 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622735/remote/smithi121/log 2024-03-29T12:00:54.174 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T12:01:12.410 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T12:01:12.620 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=9e5e4fa37553406ba62239ad0bfc8218 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T12:01:12.623 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T12:01:12.633 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T12:01:12.645 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T12:01:12.646 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T12:01:12.664 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T12:01:12.747 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T12:01:12.758 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T12:01:12.759 DEBUG:teuthology.orchestra.run.smithi098:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T12:01:12.762 DEBUG:teuthology.orchestra.run.smithi121:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T12:01:12.776 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-03-29T12:01:12.777 INFO:teuthology.orchestra.run.smithi121.stderr:bash: line 1: ntpq: command not found 2024-03-29T12:01:12.780 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T12:01:12.781 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2024-03-29T12:01:12.781 INFO:teuthology.orchestra.run.smithi121.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 18 +320us[ +320us] +/- 85ms 2024-03-29T12:01:12.781 INFO:teuthology.orchestra.run.smithi121.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 18 -108us[ -143us] +/- 48ms 2024-03-29T12:01:12.781 INFO:teuthology.orchestra.run.smithi121.stdout:^* hv03.front.sepia.ceph.com 3 6 377 18 -96us[ -131us] +/- 64ms 2024-03-29T12:01:12.781 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T12:01:12.782 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T12:01:12.782 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-03-29T12:01:12.783 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 20 +312us[ +312us] +/- 85ms 2024-03-29T12:01:12.783 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 23 -95us[ -115us] +/- 48ms 2024-03-29T12:01:12.783 INFO:teuthology.orchestra.run.smithi098.stdout:^* hv03.front.sepia.ceph.com 3 6 377 21 -110us[ -130us] +/- 64ms 2024-03-29T12:01:12.783 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T12:01:12.783 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T12:01:12.894 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T12:01:12.894 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T12:01:12.907 DEBUG:teuthology.orchestra.run.smithi098:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T12:01:12.989 DEBUG:teuthology.orchestra.run.smithi121:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T12:01:13.054 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T12:01:13.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T12:01:13.085 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T12:01:13.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T12:01:13.115 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T12:01:13.125 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T12:01:13.136 INFO:teuthology.task.internal:Duration was 1252.298580 seconds 2024-03-29T12:01:13.136 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T12:01:13.145 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T12:01:13.146 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T12:01:13.148 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T12:01:13.191 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T12:01:13.199 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T12:01:13.435 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T12:01:13.436 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi098.front.sepia.ceph.com 2024-03-29T12:01:13.436 DEBUG:teuthology.orchestra.run.smithi098:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T12:01:13.456 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi121.front.sepia.ceph.com 2024-03-29T12:01:13.456 DEBUG:teuthology.orchestra.run.smithi121:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T12:01:13.501 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T12:01:13.501 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T12:01:13.503 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T12:01:13.571 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T12:01:13.571 DEBUG:teuthology.orchestra.run.smithi098:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T12:01:13.574 DEBUG:teuthology.orchestra.run.smithi121:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T12:01:13.708 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T12:01:13.719 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T12:01:13.719 DEBUG:teuthology.orchestra.run.smithi098:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T12:01:13.746 DEBUG:teuthology.orchestra.run.smithi121:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T12:01:13.773 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T12:01:13.783 DEBUG:teuthology.orchestra.run.smithi098:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T12:01:13.790 DEBUG:teuthology.orchestra.run.smithi121:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T12:01:13.813 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = core 2024-03-29T12:01:13.838 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = core 2024-03-29T12:01:13.876 DEBUG:teuthology.orchestra.run.smithi098:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T12:01:13.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T12:01:13.896 DEBUG:teuthology.orchestra.run.smithi121:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T12:01:13.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T12:01:13.931 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T12:01:13.944 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T12:01:13.945 DEBUG:teuthology.misc:Transferring archived files from smithi098:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622735/remote/smithi098 2024-03-29T12:01:13.946 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T12:01:13.983 DEBUG:teuthology.misc:Transferring archived files from smithi121:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622735/remote/smithi121 2024-03-29T12:01:13.984 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T12:01:14.024 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T12:01:14.025 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T12:01:14.027 DEBUG:teuthology.orchestra.run.smithi121:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T12:01:14.072 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T12:01:14.086 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T12:01:14.086 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T12:01:14.097 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T12:01:14.098 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T12:01:14.101 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T12:01:14.112 INFO:teuthology.orchestra.run.smithi098.stdout: 265819 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 12:01 /home/ubuntu/cephtest 2024-03-29T12:01:14.127 INFO:teuthology.orchestra.run.smithi121.stdout: 265317 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 12:01 /home/ubuntu/cephtest 2024-03-29T12:01:14.130 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T12:01:14.141 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T12:01:14.162 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T12:01:14.188 DEBUG:teuthology.nuke:shortname: smithi098 2024-03-29T12:01:14.188 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T12:01:14.213 DEBUG:teuthology.nuke:shortname: smithi121 2024-03-29T12:01:14.213 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T12:01:14.217 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622735', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:32:11.731821', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFYgWZ1LbgwbHhpk7VTsXP9V/QXU8ImHq/fUj86pKZUcizIBTgap0pTq4nwuQdS5UBBTwAE0Tft+80B3oG7NyjA='} 2024-03-29T12:01:14.239 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622735', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:32:11.733908', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFEeDHpd8zmJPFCJ6dqE7CuXyzXFk53/XMEvbQKq08tzZzHUQC4pkdpPPyFRqyFEUI2I65wyAYFYYTc38KvJnzE='} 2024-03-29T12:01:14.268 INFO:teuthology.orchestra.console.smithi098:Power off 2024-03-29T12:01:14.268 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T12:01:14.294 DEBUG:teuthology.orchestra.console.smithi098:power off output: Chassis Power Control: Down/Off 2024-03-29T12:01:14.294 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T12:01:14.312 INFO:teuthology.orchestra.console.smithi121:Power off 2024-03-29T12:01:14.312 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T12:01:14.322 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-03-29T12:01:14.335 DEBUG:teuthology.orchestra.console.smithi121:power off output: Chassis Power Control: Down/Off 2024-03-29T12:01:14.335 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T12:01:14.360 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2024-03-29T12:01:18.323 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T12:01:18.361 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T12:01:18.435 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-03-29T12:01:18.472 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2024-03-29T12:01:22.438 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T12:01:22.474 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T12:01:22.549 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-03-29T12:01:22.584 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is off 2024-03-29T12:01:22.685 INFO:teuthology.orchestra.console.smithi121:Power off completed 2024-03-29T12:01:26.554 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T12:01:26.714 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is off 2024-03-29T12:01:26.815 INFO:teuthology.orchestra.console.smithi098:Power off completed 2024-03-29T12:01:26.920 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} duration: 1252.2985796928406 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=9e5e4fa37553406ba62239ad0bfc8218 status: fail success: false 2024-03-29T12:01:26.921 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T12:01:26.998 INFO:teuthology.run:FAIL